I'm looking to scrape data from thousands of documents stored in my company's information system. The data is organized across different tabs, with each case having a unique identification number formatted as [https://informationsystem.com/{case-identification}/general]. I specifically need to extract information from the 'General' tab of each case. To access this system, I must log in with my email and password, and I also need an authenticator token from my app for the initial sign-in. Is it feasible to write a Python script that reads a CSV file containing the case numbers and loops through the different tabs to collect the required data?
1 Answer
You can't include webpages' tabs in a CSV file like you can with Excel sheets. Just to clarify, are you sure the URLs you have correspond to the right pages? You can definitely fill in input fields and click buttons using a library like Selenium. It might be worth checking that out!
Ah, got it! So the tabs are on the web pages accessed through parts of the URL like 'general'. What I really need is a workaround because I can’t access the pages without signing in first. The login process requires an authenticator app as well, so I'm looking for a way around that.