Web Content Extraction - Science topic

You can find a self-explanatory example in the attachment :)

Hi all I finished my project, and wanted to pass on my lessons learned.

First off I was looking doing stance classification on twitter threads. This means that I had to get an entire thread, and do this in real time and a t scale since I was looking at Trump's Tweets. So I had somewhat complex and unique requirements, and no budget.

I'm also pretty much fluent in python.

That being said...

My first tendency was to grab the HTML with BeautifulSoup

https://www.crummy.com/software/BeautifulSoup/

But as I mentioned above the problem is that as mentioned in the landing page shared above by Svitlana Galeshchuk (https://github.com/Jefferson-Henrique/GetOldTweets-python) The HTML is dynamically created by as you scroll down the page so just scrapping isn't enough to get all responses (or old tweets).

Second I used the Twitter API which I discovered wouldn't always give you the full text (which is documented but easy to miss) not to mention all the other limitations of the search or streaming API.

It is worth mentioning that the besides that the streaming API is definitely worth looking at as it's very powerful. However your research needs to be about unfolding social networks (obviously), and less obviously your analysis needs to accept a semi random sample since they don't guarantee you'll see every new tweet.

This was what I used for my research because by the time I realized the issue with getting full tweet texts I had already built everything, so I grabed the full text using BeautifulSoup and the Python requests module.

Finally What I would suggest is using some sort of web simulator like Selenium

in order to "scroll down"

Then parse with BeautifulSoup .

it sounds like

GetOldTweets-python mentioned above might be a nice package that includes this, and some other solutions mention include this simulator aspect but if your trying to get all replies this is pretty much essential.

It's possible to work around using a simulator but probably isn't worth it even tough setting up a simulator often involves getting drivers to your browser and setting path variables etc. all of which are a pain to do but in my opinion probably worth it.

Where can I have more formal info about the semantic search of RG search engine?

Have a look on these articles:

Agreed with Henk... Dont confuse yourself with millions of information there o internet. Have focus on your topic, and continue search. You will soon find what are the corresponding websites that are good in particulars. And it is always good to review corresponding sites and if possible verify it from literature. 

The API have been changed a while ago. Now it is possible to mine users' profiles only using facebook apps which use an explicit permission by the user.

Hi ,

Any data type can have a label with "?". Attribute type with "?" value is most likely to be from a nominal type, hence, if you have such attribute but with string type (which is weird), simply apply "StringToNominal" filter. This will convert the string attribute into a nominal one after specifying the index of this particular attribute.

HTH.

Samer 

check this site out for datasets

Yes it is valid, I have developed an application using this technique to find out text clones, you can read my research paper to have a good vision for that

I can suggest you to use dataset from the GDELT projecthttp://gdeltproject.org/

You can retrieve the reviews through the APIs usually made available to developers. Here are some entrypoints you may use to find some info:

for TripAdvisor: https://developer-tripadvisor.com/content-api/business-content/review-content/

for Amazon: http://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_RetrievingCustomerReviews.html

for CNET: http://www.programmableweb.com/api/cnet

THANK YOU,

Text is from any form, but not handwritten and yes, there is different foreground and background color

we use python-based scrapper: http://www.crummy.com/software/BeautifulSoup/. after scrapping, perform some manual editing and get things done as per specific requirement

Have you try https://duckduckgo.com. Some of the search engine above don't work

Please have a look on http://tinyurl.com/o8ykn4y. This is the dataset used to evaluate a recent work on IE at web scale. The full description of this work and the way the corpus was extracted is described in http://www.aclweb.org/anthology/D15-1086. More information within the project's page of this group of authors http://oak.dcs.shef.ac.uk/lodie/

Hi,

 As I know, one of the main concerns in performance evaluation in machine learning techniques is that you should not use the training set for the test set too. In your case, if you use the TF/IDF or any feature vector values from your training set in your test set, your result will not be accurate enough. 

I hope it helps. 

Mostafa

 Are you doing an analysis of the text on these webpages? A lot of other pages are aggregating news from different web sources and classifying them, also news agencies like Associated Press and Reuters might be places to use? Also Google News seems to be good at indexing different news-stories. 

try visual ping

you can visually select what you want to extract

Martin Hawksey's TAGS is a great tool for harvesting tweets. It's simple to use and data is sent to a Google spreadsheet which can be downloaded as .xlsx 

It also has in built visualizations so you can see, for example, numbers of interactions and who has interacted with whom. 

I've written a little about my experience using it for the ascilite2014 conference.

I hope this helps.

ASRS Database Online 

Some demo data, including student activity logs/analytics, and grades that can be exported in CSV format are available from the "Mount Orange" Moodle Demo Site: http://moodle.com/any-way-you-want-it-moodles-demo-sites/ 

Did you mean something different, or did you want more or "real" data perhaps?

Dear Sir,

Check the file attached. May be it can help you.

Thanks

I am not sure about DBLP data set. But, if you can explore, the following link is useful to get good data set for typical analytical problems. I hope this may be of use.

Hi Phuong, you can try many algorithm:

Many tools also you can try such as Weka, Matlab, RapidMiner, etc.

Being noted that instead of using only 2 class (positive and negative) in sentiment analysis. You can consider to employ the neutral label as the 3rd class.

This article and paper could be your further reference:

Good Luck

Stopwords means the most commonly used words in particular language on which you are working. Most of the times we find 20% to 30% of text is stopwords in our normal text document. Stop words can cause problems when searching for phrases that include them. Stopwords are filtered out from the text when we process the text. and density of word is calculated as below:

Density = (frequency(word)/count(word))*100

frequency of word :- number of occurrences of word in document.

count(word) :- total number of words

Token count is the count of tokens from the text i.e. number of basic unit which you have described for your language mostly it is word.

Any word consisting of either 1 or 2 characters won't be of any significance, so we remove all of them.

To remove stopwords, we first need to detect the language. There are a couple of ways we can do this: -

Checking the Content-Language HTTP header - Checking lang="" or xml:lang=""

attribute - Checking the Language and Content-Language metadata tags If none of those are set,

You will need a list of stopwords per language, which can be easily found on the web.

Try doing it in python or R, it will be more easy for you.

- Mayur

Make a file containing the words from a sample of your pages then:

sort words.txt | uniq -c | sort -nr | head 100

The output will be the 100 most common words - this is pretty close to what you want.

Hi,

Another type of ontology is knowledge graph such as Freebase (https://www.freebase.com/), which allows users to download the weekly data dumps or use API to access the information.

best regards,

Generally, in the area of academic work, the core of programming languages are used. not software technologies.

but the decision about web-based(j2ee) or desktop-application based(j2se) should be made by through consideration of customer requirements. if you want to implement a package for classification and information extraction.

It's very hard to answer this question in general without taking into considerations what your specific needs are. Also, "Data Warehouse" is a pretty general term which basically can mean any kind of technology where you put in your data for later analysis. It can be a classical SQL database, Hadoop (yes, Hadoop can be a Data Warehouse, too), or anything else. Hadoop is a general Map Reduce framework you can also use for a lot of different tasks, including Data Warehousing, but also many other things. You also have to bear in mind that Hadoop itself is a piece of infrastructure which will require a significant amount of coding on your part to do anything useful. You might want to look into projects like Pig or Hive which build on Hadoop and provide a higher level query language to actually do something with your data.

Ultimately you have to ask yourself what existing infrastructure is already in place, how much data you have, what the kind of questions are you want to extract from your data and so on, and then use something which fits your needs.

Hi Nader,

I recommend the book "Mining the social web" (available as e-book). Here is a free chapter on Twitter: http://chimera.labs.oreilly.com/books/1234000001583/index.html

All the source code is also available on gitHub: https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition

code for Facebook: https://rawgithub.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/master/ipynb/html/Chapter%202%20-%20Mining%20Facebook.html

code for Twitter REST API: https://rawgithub.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/master/ipynb/html/Chapter%201%20-%20Mining%20Twitter.html

code for STREAMING API: https://rawgithub.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/master/ipynb/html/Chapter%209%20-%20Twitter%20Cookbook.html

A stochastic queue would be indicated. Look up queueing theory. Start here: "http://en.wikipedia.org/wiki/Queueing_theory". If you wanted to use a number of probabilistic timed automatons, you would then have the complexity of having to build in the appropriate statistical properties.

you need to create a service which fetch the information after some interval. you must know the Document structure to get the specific field or table or value. if you will have any problem with specific tool then you are welcome to ask.

Regards

Muddsair

The web content extraction is a task applied to web pages, not to databases. You are scraping unstructured data from the web, you put them in structured storage (databases) and then apply data mining algorithms to them. That's the order.

Have you considered commercial software vendors like Basis Tech?

Hi all,

thanks for your interest in my post.

We are seaching for companies interested in becoming resellers of our content extraction and management technologies and for technical people with deep expertices in web content extraction tecnologies interested in working with as software architect.