This is my first coding related project, which has grown during the years from a simple web scraper to a full stack web application. The Python library is also used by other people in their own projects, and the web application is used by myself and a few friends. After starting this project, I knew I wanted to have a career in software development.
Shopping for certain items online can be time-consuming, especially if you are a collector or looking for something rare. In this case you are probably checking multiple sites every day in hope of something worthwhile popping up. Especially Japanese sites can be a pain to navigate.
The postalservice Python library is made for scraping second hand Japanese websites. It allow easy programmatic access to the search results. Useful for creating datasets, notification systems, or web APIs. Targets the website API if available, or parses the HTML response. Certain sites require using a proxy or headless browser to avoid bot detection, and Playwright is used for that. Available on PyPI as postalservice.
In addition to the python library, the postalservice-react page serves as a frontend to that allows searching an Algolia index, which is populated by a scraper that is deployed to Oracle Cloud on a virtual machine. Algolia provides an extremely fast search experience, and the scraper updates the search results approximately every 5 minutes. This makes checking 5 sites at once a breeze, with the added benefit of an updated UI with quick preview and the size, which most of the websites lack.
Next steps for this project would be a dashboard of the scraped listings, and a similarity index to recommend similar items from the scraped listings.