Skip to content
Search

MachineLearning:Embedding

lastmod 2024-11-06created 2024-11-061 edits

Embeddings are representations of values or objects, such as text, images, and audio, designed to be consumed by machine learning models and semantic search algorithms. Embeddings translate these objects into a mathematical form according to the factors or traits each one may or may not have, and the categories they belong to.

Essentially, embeddings enable machine learning models to find similar objects. Given a photo or a document, a machine learning model that uses embeddings can find similar photos or documents. Because embeddings make it possible for computers to understand the relationships between words and other objects, embeddings are foundational for artificial intelligence (AI).

About

Embeddings are numerical representations of real-world objects that machine learning (ML) and artificial intelligence (AI) systems use to understand complex knowledge domains like humans do. For example, computing algorithms understand that the difference between 2 and 3 is 1, indicating a closer relationship between 2 and 3 than between 2 and 100. However, real-world data includes more complex relationships. For example, a bird's nest and a lion's den are analogous pairs, while day and night are opposite concepts. Embeddings convert real-world objects into complex mathematical representations that capture inherent properties and relationships between real-world data. The entire process is automated, with AI systems self-creating embeddings during training and using them as needed to complete new tasks.

Why are embeddings important?

Embeddings enable deep learning models to understand real-world data domains more effectively. They simplify how real-world data is represented while retaining semantic and syntactic relationships. This allows machine learning algorithms to extract and process complex data types and enable innovative AI applications.

Embeddings are underrated

Intro

  • Machine learning (ML) has the potential to advance the state of the art in technical writing
  • Not text generation models like Claude, Gemini, LLaMa, and GPT, but embeddings may be the ML technology with the biggest impact on technical writing
  • Embeddings are not exactly new, but they have become far more widely accessible in recent years
  • Embeddings give technical writers the ability to discover connections between texts at a scale that was previously impossible

GN⁺'s opinion

  • Embedding technology looks likely to be a big help in technical writing for identifying associations between content and discovering connections between documents. It will be especially useful in large documentation projects that have to handle a vast number of documents
  • However, embedding models still consume a lot of energy and raise ethical concerns, so careful review seems needed before adoption. Securing sufficient computing resources should be backed by efforts to minimize the environmental impact of generating embeddings
  • Providing embeddings as an API on documentation sites could be a good way to encourage creative use by the developer community. But measures for data security and privacy issues must come first
  • Startups and initiatives in technical documentation are worth actively adopting embedding technology. They could offer features that differentiate them from existing document management solutions and provide a more human-friendly experience in search, recommendations, and so on
  • Research using embeddings for document summarization, translation, topic classification, etc. is also worth looking forward to. Combining the latest natural language processing technology with embeddings could bring innovation to writing and managing technical documentation

Hacker News comments

  • Embeddings are underrated | Hacker News
  • Interesting that embeddings are the only element of modern AI that gives humans more power. This is like what Steve Jobs called "a bicycle for our minds," meaning intelligence amplification. The biggest advance in computer usability was the introduction of fast, universal local search. I use Firefox's "Find in page" feature often and use search and grep every day. Embeddings have the potential to fix search's biggest weakness by providing fuzzy search that is actually useful
  • As a documentation site owner, wondering whether to consider freely providing embeddings via a REST API or a well-known URI. It should be made clear which embedding model was used, and there is a question of whether there are embedding models suited to technical documentation
  • There are environmental concerns, but I think the public opinion that reducing AI use can solve the climate problem is wrong. For example, banning Google Maps would lead people to choose wrong routes and consume more fuel. Likewise for generating documentation with embeddings, using computing resources is more efficient
  • Shares a fun experience of how to use embeddings in a web app. Explains through documentation how to use embeddings in production
  • Embeddings seem aimless since they represent everything in the input string. Question about how to reduce the dimensionality of embeddings for a specific application. For example, when building a system to find tech support conversations, wondering how to derive embeddings that represent only the content of the conversation
  • A vector embedding is a unique summary of a document, similar to a hash code. It would be nice to have a universal standard for generating embeddings, but since it differs per AI model, it cannot have "permanence" like a hash code. There seem to be many ways to use algorithms like cosine similarity in databases and information processing apps
  • I think embeddings are underrated. The information retrieval/discovery field still uses keyword-based discovery and has not embraced semantic discovery, the modern tool. I save time by converting sentences into vector embeddings, applying k-means clustering, and summarizing with ChatGPT
  • Suggests that technical writers underrate the usefulness of embeddings. Machine learning practitioners do not underrate embeddings
  • Question about whether the OpenAI embedding model supports 8191/8192 tokens. Declaring a winner by token size is misleading; there are more important factors such as cross-lingual support and precision
  • I think embeddings are overrated; they are not the silver bullet many people hoped for. They don't fully replace simple methods like BM25 and provide only limited semantic understanding. High expectations create the belief that embeddings will retrieve exactly what you want, but unless you closely review the results, the mismatches are hard to notice

See also

Favorite site

author Zer0 sitemap