Repository navigation
Implement Logging solution #37
Description
In keeping with our avoid third party services principle, I'd like to find whether there's a good way to keep logs for easy retrieval and analysis later.
I've thought about doing this as a process running on the same VM, but that will miss startup logging and runs into limitations with combining logs from multiple instances. So we'll probably want this as a separate service.
However, I'd like to avoid signing up for an account on another service and I'd instead like to run something ourselves on Fly which I expect would be cheaper than using a hosted service and shouldn't take any more time. Fly has https://github.com/superfly/fly-log-shipper which allows for easy forwarding of logs to a running service. I think we just need to look for a good tool that allows for reviewing logs.
I would be willing to require signup for another service if it is sufficiently complex or costly to deploy and maintain our own running service.
Suggestions are welcome.
Activity
kentcdodds commentedon May 15, 2023
More info on logging with fly: https://fly.io/docs/going-to-production/monitoring/exporting-logs/#the-fly-log-shipper
Looks like the recommended solution is to use log shipper as a separate app. So now we need to decide a provider. Hopefully one we can reasonably run ourselves.
pablo-godoy-dev commentedon May 17, 2023
New Relic sounds good, they have a generous free tier without credit card required.
Integration with vector
kentcdodds commentedon May 17, 2023
Thinking about this more, I think I'd like it if our error tracking and logging service were the same. I don't know if sentry is capable of this. New Relic seems to have discontinued its error tracking service. However I think Log Rocket has both. More investigation is needed.
kentcdodds commentedon May 17, 2023
Looks like Log Rocket only handles the frontend aspect. Bummer. I'm thinking we may not be able to avoid using separate services for logging and error reporting.
phifa commentedon May 17, 2023
I often use papertrail, but that does not seem to have a free option though
pablo-godoy-dev commentedon May 18, 2023
I thought that we were talking about 2 separate services from the start. Sentry is a must-have for error tracking so I think that we need something more for log management.
We can do a sheet with all the free possibilities to analyze the pros and cons.
If I'm not mistaken, we are currently in search for:
Isn't it better to choose something from Vector's Sink list?
kentcdodds commentedon May 19, 2023
I think I'd prefer to look into the supported providers here: https://github.com/superfly/fly-log-shipper
smerchek commentedon Jun 8, 2023
We use fly-log-shipper -> Axiom - their free tier is pretty generous at 500GB for 30 days and the interface has been really great so far.
kentcdodds commentedon Jun 8, 2023
@smerchek, thank you so much for chiming in here. Their offering looks perfect actually. I'd love to see an example of this. An official example would be awesome and it'd help me a lot to visualize the changes that need to happen for something like this. If you or anyone else would have time to put something like that together that would be amazing.
Either way, I'm currently leaning in that direction.
kasperpeulen commentedon Jun 10, 2023
What about hosting something open source like the Elastic Stack on fly yourself?
All the big enterprise I have worked for self hosted the Elastic stack.
I saw this article related to Fly about it:
https://fly.io/laravel-bytes/integrating-the-elastic-stack-elk-into-a-laravel-app-on-fly/
It seems like Fly uses the Elastic Stack themself as well:
https://fly.io/blog/fly-behind-the-scenes-fresh-logging/
It is probably the cheapest way, and Kibana is really an extremely nice way to visualize incoming log data.