주혜신: BGMGenerator

Overview

생성 - Video to MIDI

• Motivation
  • 영상에 어울리는 배경음악을 생성 → 영상 편집 및 제작하는 크리에이터들에게 더 풍부한 선택지 & 즐거운 경험을 제공
• Approach
  • Transformer-based
  • 영상의 tone & mood(emotion)을 반영한 음악을 생성

1주차 (2022-04-09 ~ 2022-04-16)

Progress

• Literature Review: CMT, D2M-GAN 확인
• TikTok Vlog dataset (video-music) 만들기 위한 reference 찾기, 만들 방법 고민

To-do

• TikTok Vlog dataset (video-music) 데이터셋 제작하기 위한 영상 수집
• BGMGenerator model architecture 디자인

Literature Review

Video Background Music Generation with Controllable Music Transformer (MM 2021)

(paper) (code)

• Contribution

• Model

  • Establish three melody-music rhythmic relationships between video & BGM
    • Motion speed & Simu-note density
    • Local maximum motion saliency & Simu-note strength
    • Beat-timing encoding
  • Based on: Music Transformer(ICLR 2019, paper), Linear Transformer (ICML 2020, paper)
  • Extract rhythmic features from both video and MIDI
    • Training stage: Rhythmic features from video included
    • Inference stage: Rhythmic features from MIDI included

  

• Music Representation: MIDI

• Dataset: Lakh Pianoroll Dataset (link). video-music paired dataset 사용 X.

• Evaluation Metrics

Quantized GAN for Complex Music Generation from Dance Videos (arXiv 2022)

(paper) (code)