FFmpeg

lastmod 2025-04-23created 2014-01-05108 edits

FFmpeg is a computer program that records and converts digital audio and video streams into many kinds of formats. FFmpeg works by typing commands directly and consists of various free software and open-source libraries. Its libraries include libavcodec, an audio/video codec library used by many projects. There is also libavformat, an audio/video multiplexing and demultiplexing library. The project's name comes from the MPEG video standards group, and the "FF" in front of "mpeg" stands for "fast forward".

Library call order.

Because of dependencies between the libraries, the call order can affect compilation. The order is as follows.

libavutil.so
libavcodec.so - Includes content on encoding/decoding.
libswscale.so
libswresample.so
libavformat.so
~~libpostproc.so~~
libavfilter.so
libavdevice.so

PTS vs DTS

How to synchronize audio and video using ffmpeg libraries?

For the concepts, see Presentation timestamp#PTS vs DTS.

Syncing decoded video using FFmpeg

Syncing decoded video using Ffmpeg

To convert pts or dts to floating-point seconds, use av_q2d() on proper time_base:

// You got the context from open_input:
AVFormatContext *pFormatCtx;
avformat_open_input(&pFormatCtx, inputfilename, NULL, &format_opts);

// Get a stream from the context
AVStream pStream= pFormatCtx->streams[i];

// Convert packet time (here, dts) to seconds with:  
double seconds= (dts - pStream->start_time) * av_q2d(pStream->time_base);

// Or convert frame number to seconds with the codec context
AVCodecContext *pCodecCtx= pStream->pVideoStream->codec;
double seconds= framenumber * av_q2d(pCodecCtx->time_base);

AV_NOPTS_VALUE

FFmpeg: Timestamp specific

#define AV_NOPTS_VALUE   ((int64_t)UINT64_C(0x8000000000000000))

Undefined timestamp value.

Usually reported by a demuxer that works on containers that do not provide pts or dts.

Time Base

It is an AVRational structure made up of a numerator (num; Numerator) and a denominator (den; Denominator). It serves as the reference for time measurements.

AVStream.time_base: This is the fundamental unit of time (in seconds) in terms of which frame timestamps are represented.; For example, the formula for the duration in seconds is (AVStream.time_base.num / AVStream.time_base.den) * AVStream.start_time seconds.

Note that for AVFormatContext, the duration value refers to the AV_TIME_BASE value. ¹

AV_TIME_BASE is as follows:

AV_TIME_BASE: #define AV_TIME_BASE 1000000; Internal time base represented as integer

If the value of AVFormatContext.duration is 2166833, the actual number of seconds is 2166833 / 1000000 seconds.

Several structures (e.g. AVCodecContext) all have timestamp values, and the basis of the time_base value differs for each structure. It is best to consult the official documentation.

Getting the remaining time of a video

Stackoverflow - How to get the duration of a video in Python?

You can read the value of AVFormatContext.duration by referring to the TimeBase section above. Alternatively, there are the following methods.

opencv 0.0065 sec
ffprobe 0.0998 sec
moviepy 2.8239 sec
PyAV ???

OpenCV Duration

def with_opencv(filename):
    import cv2
    video = cv2.VideoCapture(filename)

    duration = video.get(cv2.CAP_PROP_POS_MSEC)
    frame_count = video.get(cv2.CAP_PROP_FRAME_COUNT)

    return duration, frame_count

FFProbe Duration

def with_ffprobe(filename):
    import subprocess, json

    result = subprocess.check_output(
            f'ffprobe -v quiet -show_streams -select_streams v:0 -of json "{filename}"',
            shell=True).decode()
    fields = json.loads(result)['streams'][0]

    duration = fields['tags']['DURATION']
    fps      = eval(fields['r_frame_rate'])
    return duration, fps

MoviePy Duration

def with_moviepy(filename):
    from moviepy.editor import VideoFileClip
    clip = VideoFileClip(filename)
    duration       = clip.duration
    fps            = clip.fps
    width, height  = clip.size
    return duration, fps, (width, height)

PyAV Duration

def with_pyav(filename):
    import av
    c = av.open(filename)
    return c.duration / 1000000  # `1000000` is `AV_TIME_BASE`

How to check FFmpeg log strings

char error_buffer[256] = {0,};
av_strerror(ret, error_buffer, 256);
printf("STEP #5000-5-result: %d(%s)", ret, error_buffer);

Minimize delay

I found three commands that helped me reduce the delay of live streams. The first command its very basic and straight forward, the second one it's been combined with other options which might work differently on each environment and the last command it is a hacky version that I found in the documentation It was useful at the beginning but now the first option is more stable.

Basic using -fflags nobuffer

This format flag reduces the latency introduced by buffering during initial input streams analysis. This command will reduce noticeable the delay and will not introduce audio glitches.

$ ffplay -fflags nobuffer -rtsp_transport tcp rtsp://<host>:<port>

Advanced -flags low_delay and other options

We can combine the previous -fflags nobuffer format flag with other generic options and advanced options for a more elaborated command:

-flags low_delay this codec generic flag will force low delay.
-framedrop: to drop video frames if video is out of sync. Enabled by default if the master clock is not set to video. Use this option to enable frame dropping for all master clock sources
-strict experimental, finally -strict specifies how strictly to follow the standards and the experimental option allows non standardized experimental things, experimental (unfinished/work in progress/not well tested) decoders and encoders. This option is optional and remember that experimental decoders can pose a security risk, do not use this for decoding untrusted input.

$ ffplay -fflags nobuffer -flags low_delay -framedrop -strict experimental -rtsp_transport tcp rtsp://<host>:<port>

This command might introduce some audio glitches, but rarely.

Also you can try adding: * -avioflags direct to reduce buffering, and * -fflags discardcorrupt to discard corrupted packets, but I think is very aggressive approach. This might break the audio-video synchronization

$ ffplay -fflags nobuffer -fflags discardcorrupt -flags low_delay -framedrop -avioflags direct -rtsp_transport tcp rtsp://<host>:<port>

A hacky option (found on the old documentation)

This is an debugging solution based on setting -probesize and -analyzeduration to low values to help your stream start up more quickly.

-probesize 32 sets the probing size in bytes (i.e. the size of the data to analyze to get stream information). A higher value will enable detecting more information in case it is dispersed into the stream, but will increase latency. Must be an integer not lesser than 32. It is 5000000 by default.
analyzeduration 0 specifies how many microseconds are analyzed to probe the input. A higher value will enable detecting more accurate information, but will increase latency. It defaults to 5000000 microseconds (5 seconds).
-sync ext sets the master clock to an external source to try and stay realtime. Default is audio. The master clock is used to control audio-video synchronization. This means this options sets the audio-video synchronization to a type (i.e. type=audio/video/ext).

$ ffplay -probesize 32 -analyzeduration 0 -sync ext -rtsp_transport tcp rtsp://<host>:<port>

This command might introduce some audio glitches sometimes.

The -rtsp_transport can be setup as udp or tcp according to your streaming. For this example I'm using tcp.

C Code

Try set flags of AVFormatContext to AVFMT_FLAG_NOBUFFER | AVFMT_FLAG_FLUSH_PACKETS

AVFormatContext *ctx;
// ...
ctx->flags = AVFMT_FLAG_NOBUFFER | AVFMT_FLAG_FLUSH_PACKETS;

Then try to set decoder thread to 1. It seems like more thread will cause more latency.

AVCodecContext *ctx;
// ...
ctx->thread_count = 1;

FFmpeg Deprecated Functions and Symbols

https://github.com/chelyaev/ffmpeg-tutorial

This section describes the changes made to work around deprecated functions and symbols. In some cases, a simple rename sufficed (e.g. dump_format), but in others, more significant changes to the code were required (e.g. avcodec_decode_audio2). Consult the diffs for each respective tutorial to see exactly what has changed since the original version of the tutorial.

before	after
`av_open_input_file`	`avformat_open_input`
`av_find_stream_info`	`avformat_find_stream_info`
`dump_format`	`av_dump_format`
`CODEC_TYPE_VIDEO`	`AVMEDIA_TYPE_VIDEO`
`avcodec_open`	`avcodec_open2`
`avcodec_decode_video`	`avcodec_decode_video2`
`img_convert`	`sws_scale`
`av_close_input_file`	`avformat_close_input`
`avcodec_decode_audio2`	`avcodec_decode_audio4`
`CODEC_TYPE_AUDIO`	`AVMEDIA_TYPE_AUDIO`
`url_set_interrupt_cb`	`avio_open2`
`url_ferror`	`check attribute is->pFormatCtx->pb->error`
`pstrcpy`	`av_strlcpy`

Troubleshooting

A summary of how to solve problems that can occur while using FFmpeg.

AVPacket PTS sync problem

How to get video PTS in ffmpeg

When using the PTS of an AVPacket, video sync can drift subtly. In this case, it is better to use av_frame_get_best_effort_timestamp(frame) for the PTS.

RTP: missed xxx packets

RTSP over SSL

Invalid data found when processing input

See the very next section, #Nonmatching transport in server reply.

If you are streaming, add -rtsp_transport tcp on the sending side as well.

Sending side:

ffmpeg -i video.mp4 -f rtsp -rtsp_transport tcp rtsp://localhost:8554/demo

Receiving side:

ffmpeg -rtsp_transport tcp -i rtsp://localhost:8554/demo ...

Nonmatching transport in server reply

(FFmpeg) Saving RSTP video — Enough is not enough

The following error occurs during RTSP streaming:

[rtsp @ 0x556150991900] Nonmatching transport in server reply
rtsp://address~: Invalid data found when processing input

Solve it as follows:

ffmpeg -rtsp_transport tcp -i "rtsp://address~" "001.mp4"

Could not find codec parameters for stream N : unspecified pixel format

The following error occurs during RTSP streaming after setting up tunneling with OpenSSH:

[rtsp @ 0x5dcb80d1f1c0] Could not find codec parameters for stream 0 (Video: h264, none, 640x360): unspecified pixel format
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options

Solve it as follows:

ffmpeg -rtsp_transport tcp ...

(This is a guess, but probably because RTSP uses UDP by default and queries the +1 port? .... <- needs verification)

Frame rate very high for a muxer not efficiently supporting it

(FFmpeg) Saving RSTP video — Enough is not enough

[mp4 @ 0x562101de46a0] Frame rate very high for a muxer not efficiently supporting it.
Please consider specifying a lower framerate, a different muxer or -vsync 2

As the message says, add the -vsync 2 option:

ffmpeg -rtsp_transport tcp -i "rtsp://address~" -vsync 2 "001.mp4"

Local Download

FFmpeg 3.0.1 "Einstein": 3.0.1 was released on 2016-03-29. It is the latest stable FFmpeg release from the 3.0 release branch, which was cut from master on 2016-02-14.; Ffmpeg-3.0.1.tar.gz

FFmpeg 2.8.6 "Feynman": 2.8.6 was released on 2016-02-01. It is the latest stable FFmpeg release from the 2.8 release branch, which was cut from master on 2015-09-05. Amongst lots of other changes, it includes all changes from ffmpeg-mt, libav master of 2015-08-28, libav 11 as of 2015-08-28.; Ffmpeg-2.8.6.tar.gz

FFmpeg 2.8.5: Ffmpeg-2.8.5.tar.bz2

Favorite site

FFmpeg Wiki

How to use library

Commandline help

A few FFmpeg options

How to compile

Article

FFMpeg gets a 94x performance boost with handwritten AVX-512 assembly code | GeekNews - Somewhat controversial.

Guide

Tutorials

Project

Decoding

References

Original: Duration of the stream, in AV_TIME_BASE fractional seconds ↩
Hello_world_»_FFmpeg_iOS_Video_Player.pdf ↩
How_to_Write_a_Video_Player_in_Less_Than_1000_Lines.pdf ↩
How_to_Write_a_Video_Player_in_Less_Than_1000_Lines-example.zip ↩
Demonstration_of_SDL2+FFmpeg_player_based_on.pdf ↩
FFmpeg-Mingw-compile.pdf ↩
Ffmpeg-pts-sync-error.pdf ↩
Ffmpeg_open_media_stream.pdf ↩
Ffmpeg_play.pdf ↩
Ffmpeg_transcoding.pdf ↩
Save_avframe_to_jpeg_file.pdf ↩
FFMPEG_-_H264_Decoding.pdf ↩

author Zer0 sitemap