标签：documents like text avoid object content preprocess data string

代码 :

data = pd.read_csv('asscsv2.csv', encoding = "ISO-8859-1", error_bad_lines=False);
data_text = data[['content']]
data_text['index'] = data_text.index
documents = data_text

输出

print(documents[:2])
                                              content  index
 0  Pretty extensive background in Egyptology and ...      0
 1  Have you guys checked the back end of the Sphi...      1

预处理函数

stemmer = PorterStemmer()
def lemmatize_stemming(text):
    return stemmer.stem(WordNetLemmatizer().lemmatize(text, pos='v'))
def preprocess(text):
    result = []
    for token in gensim.utils.simple_preprocess(text):
        if token not in gensim.parsing.preprocessing.STOPWORDS and len(token) > 3:
            result.append(lemmatize_stemming(token))
    return result

processed_docs = documents['content'].map(preprocess)

报错

TypeError: decoding to str: need a bytes-like object, float found

This :

processed_docs = documents['content'].map(preprocess)

is because the data frame in some cells has NaN values that can not be preprocessed, for that, you have to drop:

documents.dropna(subset = ["content"], inplace=True) # drop those rows which have NaN value cells

those unrequired rows and then apply the preprocessing.

Your data has NaNs(not a number).

You can either drop them first:

documents = documents.dropna(subset=['content'])

Or, you can fill all NaNs with an empty string, convert the column to string type and then map your string based function.

documents['content'].fillna('').astype(str).map(preprocess)

This is because your function preprocess has function calls that accept string only data type.

标签：documents,like,text,avoid,object,content,preprocess,data,string
来源： https://blog.csdn.net/Victoria_yangyu/article/details/120571715

本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享；
2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关；
3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关；
4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除；
5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。