SENTIMENT ANALYSIS OF E-COMMERCE PRODUCT REVIEWS FOR CONTENT INTERACTION USING MACHINE LEARNING

With the growth of various e-commerce applications, it has become increasingly important to understand consumer spending patterns and provide products that best respond to their needs. One way of doing this is by utilizing the data available in consumer product reviews to improve the level of content interaction. In order to tap into this data, companies can utilize a process called sentiment analysis to identify consumers’ sentiments or reactions towards certain products. This study proposes to compare different methods of sentiment analysis focused on one specific product, utilizing a machine learning approach with Naïve Bayes and Support Vector Machine classifiers with the aim of finding which method produces the best results in terms of accuracy, precision, recall and F-measure. A specific focus has been made to analyze the consumer reviews of wireless earphone products from the Indonesian e-commerce company Tokopedia. The results of this study show that utilizing the Naïve Bayes classifier, enhanced with hyperparameter tuning produced the best results in terms of accuracy, recall, and F-measure with a value of 77.03%, 73.03%, and 74.41% respectively. Whereas the best precision was obtained by utilizing an SVM classifier enhanced with hyperparameter tuning, producing a value of 76.05%.


Introduction
The use of e-commerce applications has become a common phenomenon in today's society.E-commerce, which stands for electronic commerce can be defined as, "the use of the internet as the basis for implementing various business processes" (Simon & Shaffer, 2001).In Indonesia, there are many ecommerce-based companies that have been developed in various fields of general needs such as online buying and selling, ticket transactions, and online transportation facilities.These companies also compete with each other in order to get customers who use ecommerce applications.There are many ways to attract the attention of customers, such as providing large discount promotions.
However, for e-commerce in the form of an online shopping platform, consumers are not only pursuing discounts.For online buying and selling platforms, the shopping content and product categories provided are also important factors that drive customers towards certain ecommerce applications; this is also known as ‗content interaction'.For example, the variety of the shopping catalogue offered by the ecommerce company or because of the guaranteed originality of the goods sold.
Another important feature of ecommerce platforms are its internal product review section.Once customers have purchased goods from an e-commerce platform, they are usually prompted to leave a review of the product, which can cover everything from the product quality, value for money, shipping CAKRAWALA -Repositori IMWI | Volume 6, Nomor 1, Februari 2023 p-ISSN: 2620-8490; e-ISSN: 2620-8814 208 speed, and seller interaction.This internal review system not only helps to attract customers by providing insight into the product quality offered by the platform but also provides an important source of data for the ecommerce company, which it can use to enhance the shopping experience for its customers.
In order to tap into this data, companies can utilise a process called sentiment analysis to identify consumers' sentiments or reactions towards certain products as shared in the platform's product review section.Sentiment analysis, also known as opinion mining, is a field of study that analyses people's opinions, sentiments, evaluations, appraisals, attitudes, and emotions to entities such as products, services, organizations, individuals, problems, events topics, and their attributes (Sari et al., 2018).Once sentiment analysis has been done, the resulting data can be used by companies to make better decisions such as highlighting certain popular products or pulling negatively reviewed products or sellers from the main search results page.
In this study, the author aims to address the following research question: -In implementing sentiment analysis for certain product reviews using a machine learning approach, which classification method between Naïve Bayes and Support Vector Machine, has the best performance when measured in terms of accuracy, precision, recall, and F-measure?‖.The author will utilise sentiment analysis to analyse a case study of consumer reviews of ‗wireless earphone' products from the ecommerce company Tokopedia.Tokopedia is one of Indonesia's largest e-commerce companies which currently has over 100 million active monthly users and 10 million merchants (Tokopedia, 2021).It is best known for its good reputation, low product prices, and large product variety (Romadhon, 2021).The author will utilise data obtained from product reviews and comments posted in the company's internal review section.
The data will go through a preprocessing stage and will then be classified using the Naïve Bayes and Support Vector Machine classifiers.By comparing the levels of accuracy, precision, recall, and F-measure of these two frequently used machine learning classification methods, for specific product reviews, it is hoped that a more in-depth analysis of product review data will be produced.Thus, the results of sentiment analysis with the best performance can be utilized by e-commerce companies and their sellers to improve their consumer shopping experience for that specific product.The sentiment analysis model used in this study can also be applied more broadly by other ecommerce companies to analyze whatever type of product they choose.

Method 1. Preliminary Study
As companies gain access to large amounts of data that were not accessible to them in the past, knowing how to analyse and utilize this data has become more important than ever.Sentiment analysis is one of the ways that researchers and companies can analyse data that is presented in a textual format, rather than a basic numerical format (Sharma, 2020).In this regard, data can be collected in the form of customer reviews, complaints, discussions, opinions, tweets, or other natural language sources (Vanaja & Belwal, 2018).
Sentiment analysis can be conducted using different algorithms, however there are two approaches that are most commonly used, i.e. lexicon-based approaches (Taboada et al., 2011), which often uses a dictionary-based approach to extract and annotate texts with a sentiment value (Jurek et al., 2015), or machine learning approaches such as Naïve Bayes, Decision Trees (Medhat et al., 2014), (Suresh & Bharathi, 2016), and Support Vector Machines.The Naïve Bayes classifier is one of the most commonly used machine learning approaches and works by counting the probability of each aspect within a text, utilising the Naïve Bayes Theorem (L eu ng, 2007).Whereas Support Vector Machines work by finding the optimal hyperplane within a labelled training dataset, where data is clearly classified ( H a n d a y a n i & H a k i m , 2 0 2 0 ) .
Although it is not always easy to do, analysing data through sentiment analysis can provide a wealth of information and often provide more insight than basic numerical data.As such, several studies have been done in this field which explore the different methods of doing sentiment analysis.
In a study (Marong et al., 2020), titled -Sentiment Analysis in E-Commerce: A Review on The Techniques and Algorithms‖, they review several different methods of sentiment analysis, such as lexicon-based and monitored machine learning.Monitored machine learning approaches include the Naïve Bayes method, which is commonly used in the retail industry, whereas the lexicon-based method utilizes predefined word phrases and opinion idioms where every phrase and idiom is assessed as either positive or negative sentiment.Their article concluded that sentiment analysis can be a very useful tool for companies to enhance their user experience, however there are still many challenges in implementing it, such as the use of sarcastic reviews which can be difficult to automatically classify as positive or negative.
Another study (Sari et al., 2018), titled -Measuring E-Commerce service quality from online customer review using sentiment analysis‖, utilized sentiment analysis to assess user reviews of the e-commerce company Tokopedia, as shared on the TrustedCompany.comwebsite.Using the Naïve Bayes classification, they classified the reviews into positive and negative sentiments for five dimensions of electronic service quality, which includes web design, reliability, responsiveness, trust, and personalization.Their results showed that Tokopedia scored positively in the trust and web design dimensions but scored negatively in the personalization and reliability dimension.
A third study (Hariawansah, 2018), titled -Sentiment Analysis in E-Commerce using Naïve Bayes Method‖, utilized sentiment analysis to compare the public sentiments towards e-commerce companies such as Lazada, Bukalapak, and Tokopedia.The author compiled the reviews from each of the company's Facebook pages and utilized the Naïve Bayes method to classify negative and positive comments from each company's users.The results of the study revealed that Tokopedia and Lazada had the best review, whereas Bukalapak had more negative sentiments.The Naïve Bayes Classifier was very useful in this study to classify negative and positive comments, which helped provide an easily understandable but accurate review of each company based on its respective users' experiences.
Another study (Noor & Islam, 2019), titled -Sentiment Analysis for Women's E-Commerce Reviews using Machine Learning Algorithms‖, used the Waikato Environment for Knowledge Analysis (Weka) software of machine learning algorithms to conduct its sentiment analysis of women's fashion products sold on the e-commerce platform, Amazon.The authors used four different categories (Bayes Theorem, Rules, Trees, and Support Vector Machines) of classifiers along with data pre-processing functions, feature extraction and attribute selection.The results of the study showed that SVM classifiers provided the most accurate results out of the four classifiers, and the authors also recommended In another study (T.Willianto, Supryadi, 2020), titled -Sentiment Analysis on E-Commerce Product Using Machine Learning and Combination of TF-IDF and Backward Elimination‖, the authors applied sentiment analysis to consumer product reviews in ecommerce platforms in Indonesia with a more optimized method.The method tested in this study was the Terms Frequency -Inverse Document Frequency (TF-IDF) method for feature extraction, Backward Elimination for the feature selection stage, and a comparison between five types of classifiers namely Naïve Bayes, K-Nearest Neighbor, Decision Tree, Random Forest, and Support Vector Machine.
From the results of this study, it was found that the best level of accuracy was achieved by using the TF-IDF method, Backward Elimination, and then using SVM as a classifier.It was also found that using the Backward Elimination method at the feature selection stage helped increase the accuracy of the five types of classifiers in conducting sentiment analysis.
Lastly, in a study, titled "Sentiment Analysis of Indonesian E-Commerce Product Reviews using Support Vector Machine Based Term Frequency Inverse Document Frequency", the researchers used the CRISP-DM or Cross-Industry Standard Data Mining Process method and compared the performance of four machine learning algorithms, namely Random Forest, Gradient Boost, Support Vector Machine, and Decision Tree, in conducting sentiment analysis on product reviews.on e-commerce platforms in Indonesia.The results of this study indicate that the machine learning algorithm that has the highest level of accuracy is the use of the Support Vector Machine, with an accuracy of 95.87%.These results indicate that the SVM algorithm can be used as a reliable method to perform an in-depth analysis of consumer behavior delivered through product reviews on e-commerce platforms.
ased on the review of related literature, it can be seen that there are various ways and methods to perform sentiment analysis, either through a machine learning approach or other approaches such as a lexicon-based approach.From these previous studies, there are several studies that use a machine learning approach with various classifiers such as Naïve Bayes, Decision Trees, Support Vector Machines, and others.Although there have been several studies comparing the effectiveness of these various machine learning classification methods, often these studies apply sentiment analysis to all types of products available on an e-commerce platform or to consumer ratings of the ecommerce platform as a whole.There have been no studies that have focused on comparing the effectiveness of machine learning classification methods, such as Naive Bayes and Support Vector Machines, on consumer reviews of a particular type of product.In fact, a more in-depth analysis like this can yield more specific insights not only for e-commerce companies as a whole, but also for sellers of these products.

Research Framework
The research that was conducted was initiated from the writer's interest in the existing data in many e-commerce companies' internal product review sections.It is undeniable that there is a lot of data on these pages and this can help us in analyzing products and customers who buy or use the goods/services that the company provides.Therefore, through this study, the author wants to create a sentiment analysis of the reviews left by consumers in the e-commerce company's product review section, which can then help companies make better decisions to improve the customers' shopping experience.
The research framework of this study can be seen in Figure 1: As shown in the research framework in Figure 1, the first step that must be taken is to identify the research problem.The problem seen here is that there is a lot of sentiment analysis done on product reviews, but there is no specific focus on the type of product being reviewed.There have not been many studies that utilise sentiment analysis to review one specific product, even though this can help many e-commerce companies and their sellers see in more detail the sentiment towards a particular product.
The next step is a literature study, this stage is used to understand the current situation and trends.This stage is done by looking for information and references that are relevant to the topic or problem that is at the focus of this study.
Next is to define the scope.The scope that the researcher uses in this study is the Tokopedia marketplace and the search keyword "wireless earphone".This is done to obtain valid product reviews that are still within a predetermined period of time, so as not to go beyond the scope of the research.The author chose wireless earphones as the focus of this research because it is a product that is becoming increasingly important in the digital era, but not many consumers are aware of this product.This is because many newly released cellphones are starting to be produced without earphone jacks, thus making traditional wired earphones obsolete.This focus on only one type of product is intended so that the dataset collected is specific for one type, and expects accurate sentiment results for one type of product.
The next step is to filter the products.This study filtered the six most popular wireless earphones currently sold on Tokopedia, where each product has around 900 reviews.In the next step, the writer will start the data collection process, which includes the collection of training data.This data collection CAKRAWALA -Repositori IMWI | Volume 6, Nomor 1, Februari 2023 p-ISSN: 2620-8490; e-ISSN: 2620-8814 212 process will be carried out using the Web Scraper method directly from the Tokopedia platform.
The next stage, namely pre-processing of data is needed to clean the data from noise, ensure uniformity of word types and reduce word volume.This pre-processing stage will allow the classification process to work more smoothly and efficiently in its calculations.Before the results of data pre-processing are entered into the classification algorithm, data training will be carried out to perform attribute selection.In this phase, each word will be sorted by word frequency.
The classification stage is the main stage for understanding sentiment patterns from reviews that have gone through the preprocessing stage.Classification in this study will be carried out using the Naïve Bayes theorem and SVM and trying to assess positive, neutral, or negative sentiment in a review.And the last stage is evaluation.The evaluation that the researcher carried out was by using the Confusion Matrix to process the classification results and determine the values of accuracy, precision, recall, and F-Measure.

Proposed Methods
Although different algorithms may be used to conduct sentiment analysis, the methods used to prepare and analyse the data typically follows a similar process, as explained below (Bayhaqy et al., 2018), (Zainuddin & Selamat, 2014).

Filter Product
The first step is to filter the products whose reviews will be analysed.This filter product step was done using a feature of the Tokopedia platform by inserting the keywords -wireless earphone‖ in the search bar and sorting the results based on the greatest number of reviews.The author limited the results based on the criteria that the top ten selected products must be distinct and cannot be the same products.-

Data Collection
During the data collection phase, the author compiled the available customer review data for each selected product that has gone through the initial filter.Of the 6 most popular wireless earphones currently sold on Tokopedia, where each product has around 900 reviews, this resulted in around 4,500 reviews that were analysed.Data was collected using the Web Scraper program directly from the Tokopedia platform.This Web Scraper program is a Google Chrome extension that functions to scrape the required data.The result was produced in the form of a file that can be extracted in an excel format which can then go through the next phase which is data preprocessing.

1.2.3.
Data Pre-Processing In the next stage, what needs to be done is to carry out the data pre-processing process.The product review data that has been collected using Web Scraper underwent a data preprocessing prior to sentiment analysis.The preprocessing stages can be seen in the image below.Converting Emoticons An emoticon is a representation of a human facial expression using only keyboard characters such as letters, numbers, and punctuation marks.There is also an emoji, which is an image small enough to insert into text that expresses an emotion or idea.The word emoji essentially means -picture-character‖ (from Japanese e --picture,‖ and moji --letter, character‖).In this stage, the reviews that have emoticons were converted into its appropriate string.The library that was used for these emojis and emoticons are the following github repository: https://github.com/NeelShah18/emot.

ii.
Cleaning One of the first steps in data pre-processing is cleaning.Cleaning in this phase means the process of removing or erasing symbolic components that are present in the review data.This is done because these components do not have any effect on the sentiment value of a review.The identification of these components in the cleaning phase was done using regular expressions, as shown in the table below.Case Folding The next step in data pre-processing is case folding.Case folding is the process of converting words to lowercase letters.The purpose of changing words to lowercase is to convert case sensitive to case insensitive (there is no effect when using uppercase or lowercase letters). ii.

Removing Excessive Words
Sometimes many words are given excessive affixes by adding a few more letters in front or behind the sentence.In this process, the letters are omitted so that there is less variation in the data. iii.

Removing Short Words
This process cleans up short words that have no meaning, such as -a‖, -z‖, -yg‖ etc. iv.

Stemming
Stemming is the process of converting words into basic tenses.In this study, the library from https://github.com/sastrawi/sastrawiwas used.With the help of this library, it can make it easier for us to get basic words.

Removing Null Data
After the data goes through the cleaning process, many rows of data will have a Null value, or have no review.This data will be deleted from the dataset list so that it does not add empty sentiment space later. vi.

Filtering Stop words
Filtering stop words is the stage of taking important words using a stop list algorithm (removing less important words) or wordlist (saving important words).Stop words are common words that usually appear in large numbers and are considered meaningless.Examples of stop words in Indonesian are -yang‖, -dan‖, -di‖, -dari‖, etc.The meaning behind using stop words is that by removing low-information words from a text, we can focus on the important words instead.The Indonesian stop word dictionary used in this study was https://github.com/stopwordsiso/stopwords-id.

Feature Extraction
Feature extraction in this study was carried out using the RapidMiner application.The feature extraction used is TF-IDF, where after the dataset the attributes and roles are determined to be processed into Process Documents from Data (Qaiser & Ali, 2018).In Process Documents from Data there are several processes.Process Tokenize breaks the dataset into tokens of one word each.Then there is the n-Grams process, where this process helps group tokens based on the number of n in n-Grams.This makes it easier to group specific words together, so that long review datasets can calculate vectors in a scalable way.

Distribution of Dataset Portions
After the extracted data is cleaned through pre-processing, the next step is to divide the data into two parts, where the first part is used for training and the second part is used for testing.The ratio of the distribution of this dataset portion is 70% for training and 30% for testing.This training data will later be used for machine learning needs and the test data used for evaluation purposes.

Classification
There are many algorithms that can be used for classification purposes.In classifying sentiments in this study, two algorithms were tested to achieve the best results.The classification algorithms used include: SVM and Naive Bayes.Each algorithm produced a CAKRAWALA -Repositori IMWI | Volume 6, Nomor 1, Februari 2023 p-ISSN: 2620-8490; e-ISSN: 2620-8814 215 classification model that has been trained using training data.
In the data training process, a hyperparameter tuning process was carried out, where the hyperparameter values will be determined, one model will be optimized for each set, and the model which produces the most accurate output will be selected (Wu et al., 2019).This tuning is also tested several times using cross validation (Berrar, 2019), where the data will be divided into two types of data, namely training data which will be used to compare with the distance of the test data, and test data which will be used for the classification process at the testing stage.This process is carried out according to the given K-Fold value, where if the Fold is 10, then this data will be tested 10 times.

Results and Discussion
Once the data has gone through the stages of pre-processing, sentiment analysis was done utilizing Naïve Bayes and SVM.At this classification stage, for each classification method used, hyperparameter tuning was performed first.The purpose of this hyperparameter tuning is so that the accuracy value obtained can be much better, and this tuning is done using a grid search process (Pannakkong et al., 2022).
After the hyperparameter value with an accuracy level was found, then the hyperparameter was used to conduct test data training.After the test data had been determined, a performance test was carried out on the testing data.The final result of this classification is a 3 x 3 confusion matrix mapping where there are positive, negative, and neutral classes (S.Visa, B. Ramsay, A. Ralescu, 2011).

Naive Bayes
The initial stage of Naïve Bayes classification was done by performing hyperparamater tuning.This process was carried out using test data which comprised of 70% of the research dataset portion.The parameters in the grid search process that were tuned here include the value of the laplace correction which is either true or false, and the value of the number of kernels which is given a range from 1 to 100.This tuning was also carried out with cross validation, where the test was carried out 10 times so that the results obtained are the best.The results of this hyperparameter tuning can be seen in Table 2. Based on the results of the tuning process above, it was found that the 8 th iteration was where the hyperparameter laplace correction was false, and the number of kernels 31 resulted in the highest accuracy rate of 75.17%.

Table 2. Results of Naïve Bayes
After the hyperparameter was found, the performance test with test data and data testing can be continued.The results of the Confusion Matrix for Naïve Bayes with hyperparameter tuning obtained a True Positive of 1036.

Support Vector Machine
In the Support Vector Machine classification, the hyperparameter tuning performed was the kernel cache parameter with a value between 0 to 100 and a C value from -1 to 1.
This tuning was also carried out with cross validation, where the test was carried out 10 times so that the results obtained are the best.The results of this hyperparameter tuning can be seen in Table 3. Table 3 Based on the results of the tuning process above, it was found that the 13th iteration was where the kernel cache hyperparameter is worth 10 and the value of C -1 produces the highest accuracy rate of 72.65%.After the hyperparameter was found, the performance test with test data and data testing can be continued.The results of the confusion matrix for SVM with hyperparameter tuning obtained a True Positive of 995.

Discussion
In this study, two experiments have been carried out to find out whether using Naive Bayes and SVM classifiers, assisted by hyperparameter tuning was capable of finding the best sentiment results with datasets that are only intended for specific products of an ecommerce marketplace.This experiment includes: • Naive Bayes • SVM From the results of the confusion matrix that has been created, it is hoped that it can measure aspects of this dataset.In this confusion matrix, accuracy measures the similarity between predicted product reviews and actual product reviews, while precision measures the fraction of all detected product reviews that are predicted to be product reviews, addressing the important issue of identifying which reviews fall into certain categories.The higher the precision and accuracy values, the better the performance.

Key Findings
In this study, sentiment analysis has been conducted on specific e-commerce product reviews with the keyword -wireless earphones‖, where the dataset used only includes reviews of these types of goods, using machine learning with different types of algorithms, i.e.SVM & Naïve Bayes with a total of 4500 product reviews consisting of 3 classes of positive reviews, neutral reviews, and negative reviews.Before being analyzed, the data has also gone through the process of tokenization, case folding, normalization, filtering, stop word removal, stemming, and TF-IDF.
The results of this study indicate that the process of sentiment analysis for a product specification is also very possible to do.Product sentiment analysis is carried out by calculating the weight of positive, neutral, and negative sentiments contained in product reviews in Indonesian.After doing the classification process through the preprocessing stage, sentiment analysis, and calculating word weights with TF-IDF.Based on the weight, it will approach the word with test data and training data.With the process of analyzing the words in the product review, it will be able to analyze the product review sentiment contained in the word.The use of hyperparameter tuning as well as confusion matrix mapping also helps to get the required results.Based on the results of research sentiment analysis using the Naïve Bayes method, an accuracy of 77.03% can be obtained.
Based on these results, it can be concluded that this modeling can be used to support the detection of product reviews in Indonesia today.It is hoped that this model can also be applied so that it helps e-commerce companies and sellers in e-commerce platforms to conduct a more in-depth analysis of customer spending habits as expressed by their reviews on the platform.

Recommendations
Based on the conclusions obtained in this study, the following are some suggestions to improve the research on sentiment analysis in the future: 1.Using a larger dataset.

Figure 2 .
Figure 2. Data Pre-Processing Stages i.Converting Emoticons An emoticon is a representation of a human facial expression using only keyboard characters such as letters, numbers, and punctuation marks.There is also an emoji, which is an image small enough to insert into text that expresses an emotion or idea.The word emoji essentially means -picture-character‖ (from Japanese e --picture,‖ and moji --letter, character‖).In this stage, the reviews that have emoticons were converted into its appropriate string.The library that was used for these emojis and emoticons are the following github

Table 1 .
Symbolic Components to be Removed Symbol

Table 4
In Table4, it can be concluded that the Naïve Bayes method has the highest accuracy rate, which is 77.03%.The highest precision was obtained by SVM with a value of 76.05%.The highest recall was obtained by Naïve Bayes with a value of 73.03%.And the highest F-Measure was obtained by Naïve Bayes with a value of 74.41%.