PyTorch vs Tensorflow
lastmod 2021-12-28created 2021-12-283 edits
Common
People usually say TensorFlow is for industry and PyTorch is for research, but that is based on outdated information
Practical considerations: usually comes down to 3 comparisons
- Model availability: how many models are publicly available
- Deployment infrastructure: how efficiently can models be deployed
- Ecosystem: various environments/hardware
Model Availability
Both PyTorch and TensorFlow have their own official model repositories
HuggingFace
- 85% are available only in PyTorch, and 50% of the non-exclusive models are also available. Only 16% of all models are available in TensorFlow (only 8% TF-only)
- For the Top 30 most popular models, all are available in PyTorch, but only 18 of them are available in TensorFlow
Research papers
- PyTorch usage in papers submitted each year keeps increasing
- Starting from 7% in 2017, it reached almost 80% in 2021
- Many chose PyTorch because of the difficulties of TensorFlow 1
- These difficulties were largely improved in TensorFlow 2, released in 2019, but PyTorch had already established itself in research
- 55% of researchers who used TensorFlow in 2018 moved to PyTorch in 2019
- 85% of researchers who used PyTorch in 2018 still used PyTorch in 2019
According to Papers With Code data
- Tensorflow keeps declining, while PyTorch and other frameworks are increasingly used
- This trend continued even after the release of TensorFlow 2. Not enough to reverse the difficulties experienced with TF1
Results
In conclusion, PyTorch dominates the market in terms of model availability
Other exceptions:
- GoogleAI mainly used TensorFlow, but Google Brain uses JAX/Flax
- DeepMind also adopted JAX and developed Haiku, a neural network library for Jax (Sonnet for JAX)
- JAX: a separate framework made by Google. Gaining popularity in research
Deployment
TensorFlow was optimal for deployment-oriented applications from the start
- With TensorFlow Serving, TensorFlow Lite, etc., can be deployed painlessly to cloud/servers/mobile/IoT devices, etc.
PyTorch was very weak in terms of deployment, but in recent years has worked to close this gap
- TorchServe and PyTorch Live
In conclusion, TensorFlow is still better in terms of deployment
- In particular, being able to use TFLite locally and on Coral devices is essential for other industries too
- PyTorch Live focuses only on mobile, and TorchServe is still in an early stage
Ecosystems
PyTorch
- Official PyTorch Hub exists: has various models for Audio/Vision/NLP, etc.
- SpeechBrain: official open-source speech toolkit, with many features such as ASR/speaker recognition, authentication
- Various tools and libraries, including Fast.ai
- TorchElastic: AWS+Facebook collaboration project
- TorchX: library for fast machine learning development and deployment
- PyTorch Lightning: Keras of PyTorch
TensorFlow
- Official TensorFlow Hub: various models including BERT
- Model Garden: source code repository for SOTA (State-of-the-ART) models
- TensorFlow Extended (TFX): E2E platform for model deployment
- Vertex AI: Google Cloud's Unified machine learning platform
- Media Pipe: multimodal, cross-platform framework for building machine learning pipelines for features such as face recognition, hand tracking, and object recognition
- Coral: toolkit built to support products (IOT) with built-in local AI
- TensorFlow.js: JavaScript library for machine learning
- TensorFlow Cloud: connects a local development environment to Google Cloud
- Colab: cloud-based notebook environment (similar to Jupyter)
- Playground: educational tool good for understanding neural network basics
- Datasets: stores datasets regularly released by Google (also available to PyTorch users)
Results
In conclusion, TensorFlow has the better ecosystem
- Google has invested heavily over the years
- Things like connecting to Google Cloud or easy porting to Coral devices are also better in some fields
So which one should you use?
If you are in industry, the choice may differ depending on which industry you are in (the original article provides a flowchart for an easy choice)
- If you use deep learning in an industrial setting, you are probably already using TensorFlow and can stick with it
- PyTorch may also be worth considering because of TorchServe
- For mobile, use PyTorch Live; for audio/video input, TensorFlow
- If using AI in IoT, TensorFlow Lite + Coral
If you are a researcher?
- You are probably already using PyTorch and can stick with it
- As a special exception, consider TensorFlow if you do Reinforcement Learning
- If you don't want to use TensorFlow, JAX is also worth considering
- Whatever you choose, keep an eye on JAX in 2022
If you are a professor?
- May depend on the goals of the course
- To produce engineers who will use machine learning right away in the field: TensorFlow
- To understand deep learning and deep learning models: PyTorch
If you are changing careers?
- Both PyTorch / TensorFlow are good options
- Choose whichever is easier to use
- If you are not tied to a particular framework, TensorFlow, the industry standard, would be better
- TensorFlow was better for job hiring/job seeking
If you do deep learning as a hobby?
- Depends on what you want to do, but basically PyTorch is better
If you are a complete beginner to deep learning?
- Starting with Keras is recommended
- If you want an environment similar to native Python, PyTorch, especially PyTorch Lightning