Having trouble with the object of type bytes is not JSON serializable error in Python? Don’t worry; we are here to help. Let’s discuss why this issue occurs and how to overcome it.

What Causes object of type bytes is not JSON serializable Error And How To Fix It

The error object of type bytes is not JSON serializable in Python might occur once you attempt to convert bytes object to JSON string.

An illustration of the error can be seen in the following.

from scrapy.spiders import Spider

from scrapy.selector import Selector

from w3school.items import W3schoolItem

class W3schoolSpider(Spider):

name = "w3school"

allowed_domains = ["w3school.com.cn"]

start_urls = [

 "http://www.w3school.com.cn/xml/xml_syntax.asp"

]

def parse(self, response):

sel = Selector(response)

sites = sel.xpath('//div[@id="navsecond"]/div[@id="course"]/ul[1]/li')

items = []

for site in sites:

item = W3schoolItem()

title = site.xpath('a/text()').extract()

link = site.xpath('a/@href').extract()

desc = site.xpath('a/@title').extract()

item['title'] = [t.encode('utf-8') for t in title]

item['link'] = [l.encode('utf-8') for l in link]

item['desc'] = [d.encode('utf-8') for d in desc] items.append(item)

return items

In the above example, you might notice that we have created bytes objects. 

However, the bytes objects are not supported by this method by default. Thus, we will remove the call to the method encode() or utilize the function decode() to convert this bytes object to the string to fix the issue.

item['title'] = [t.encode('utf-8') for t in title]

item['link'] = [l.encode('utf-8') for l in link]

item['desc'] = [d.encode('utf-8') for d in desc]

items.append(item)

Each of the d.encode(), l.encode(), and t.encode() calls makes a string bytes. Avoid doing this; instead, let the JSON format serialize these.

Plus, you won’t need to encode too much when it is not necessary. You can leave the encoding to the module json module and your standard file object given by the method open().

Here is an example.

class W3SchoolPipeline(object):

def __init__(self):

self.file = open('w3school_data_utf8.json', 'w', encoding='utf-8')

def process_item(self, item, spider):

line = json.dumps(item) + '\n'

self.file.write(line)

return item

The Bottom Line

We have covered everything about the cause and solution to fix the object of type bytes is not JSON serializable error in Python. If you ever encounter this error, just employ our method above, and you are good to go.

Besides this issue, we also provide a guide on how to fix TypeError: can only join an iterable, a quite common problem when working in Python. So if you are struggling with this error, check out our post to get it solved.