Download Personal media broadcasting system

US007877776B2
(12) Ulllted States Patent
(10) Patent N0.:
Krikorian et al.
US 7,877,776 B2
(45) Date of Patent:
Jan. 25, 2011
(54)
PERSONAL MEDIA BROADCASTING
SYSTEM
5,602,589 A
5,610,653 A
5,661,516 A
2/1997 Vishwanath et a1.
3/1997 Abecassis
8/1997 Carles
(75)
Inventors: Jason Krikorian, San Mateo, CA (US);
5,666,426 A
9/1997 Helms
Blake Krikorian, San Mateo, CA (US)
(73) Assignee: Sling Media, Inc., Foster City, CA (U S)
(*)
Notice:
(Continued)
Subject to any disclaimer, the term of this
patent is extended or adjusted under 35
U.S.C. 154(b) by 1424 days.
FOREIGN PATENT DOCUMENTS
CN
1464685
12/2003
(21) App1.No.: 11/147,664
(22) Filed:
Jun. 7, 2005
(65)
(Continued)
Pl‘lOl‘ Publication Data
Us 2006/0095471 A1
OTHER PUBLICATIONS
May 4’ 2006
ReplayTV 5000 User’s Guide, 2002, entire document.*
Related US. Application Data
(60)
Prov1s1onal appl1cat1on No. 60/577,833, ?led on Jun.
Primar ExamineriHumer B‘ Lonsbe
7 2004
’
(51)
(Continued)
y
'
“y
74 Allorn 6)’, A genl, 0r Firmilng rassia Fisher & Lorenz,
RC.
Int. Cl.
H04N 7/18
H04N 7/173
(2006.01)
(2006.01)
(57)
(52)
US. Cl. ......................... .. 725/95; 725/80; 725/133;
(58)
_
_
_
Field of Classi?cation Search ..... ......... .. 725/80,
S
1,
_
?l f
1 725
141’ 93*97
725/ 141
ee app lcanon
e or Comp ete Seam
References Cited
(56)
ABSTRACT
.
.
.
.
.
A personal med1a broadcastmg system enables v1deo d1str1
bution Overa computernetwork and allows auserto View and
control media sources over a computer network from a
lstory'
remote location. A personal broadcaster receives an input
from one or more types of media sources, digitiZes and com
presses the content, and streams the compressed media over a
computer netWork to a media player running on any of a Wide
U.S. PATENT DOCUMENTS
3,416,043 A
12/1968 Jorgensen
range of client devices
for viewing the media. The“ system
_
4 254 303 A
3/l98l Takizawa
may alloW the user to 1ssue control commands (e.g., channel
5,161,021 A
11/1992 Tsai
up”) from the media player to the broadcaster, causing the
5,237,648 A
8/l993 Mills et 31
source device to execute the commands. The broadcaster and
5,386,493 A
5’434’590 A
M1995 D eg en et Al
7/l995 Dinwiddie Jr et a1
the media player may employ several techniques for buffer
ing, transmitting, and vieWing the content to improve the
5,434,678 A
7/1995 Abecassis
user 5 expenence
5,438,423 A
8/1995 Lynch et a1.
5,493,638 A
2/1996 Hooper et a1.
52 Claims, 2 Drawing Sheets
Central
Server
m
A/V Source
Devices
Q
Home Internet
C ‘
Jig
,
Personal
Broadcaster
OmNetwork
m
Storage
Device
1_3_0
Local Clients
Remote Clients
E
E1
US 7,877,776 B2
Page 2
US. PATENT DOCUMENTS
5,682,195
5,684,918
5,696,869
5,706,290
5,708,961
5,710,605
5,722,041
5,757,416
5,774,170
5,778,077
5,794,116
5,822,537
5,831,664
5,850,482
5,852,437
5,880,721
5,892,536
5,898,679
5,909,518
5,911,582
5,922,072
5,936,968
5,953,485
5,968,132
5,987,501
6,002,450
6,008,777
6,014,694
6,020,880
6,031,940
6,036,601
6,040,829
6,043,837
6,049,671
6,075,906
6,088,455
6,088,777
6,091,886
6,097,441
6,104,334
6,108,041
6,115,420
6,117,126
6,141,059
6,141,447
6,151,444
6,160,544
6,201,536
6,208,805
6,212,282
6,222,885
6,223,211
6,240,459
6,240,531
6,243,596
6,256,019
6,263,503
6,279,029
6,282,714
6,286,142
6,340,994
6,353,885
6,356,945
6,357,021
6,370,688
6,389,467
6,408,128
6,434,113
6,442,067
6,456,340
6,466,623
10/1997
11/1997
12/1997
1/1998
1/1998
1/1998
2/1998
5/1998
6/1998
7/1998
8/1998
10/1998
11/1998
12/1998
12/1998
3/1999
4/1999
4/1999
6/1999
6/1999
7/1999
8/1999
9/1999
10/1999
11/1999
12/1999
12/1999
1/2000
2/2000
2/2000
3/2000
3/2000
3/2000
4/2000
6/2000
7/2000
7/2000
7/2000
8/2000
8/2000
8/2000
9/2000
9/2000
10/2000
10/2000
11/2000
12/2000
3/2001
3/2001
4/2001
4/2001
4/2001
5/2001
5/2001
6/2001
7/2001
7/2001
8/2001
8/2001
9/2001
1/2002
3/2002
3/2002
3/2002
4/2002
5/2002
6/2002
8/2002
8/2002
9/2002
10/2002
Hendricks et al.
Abecassis
Abecassis
Shaw et al.
Hylton et al.
Nelson
Freadman
Birch et al.
Hite et al.
Davidson
Matsuda et al.
Katseff et al.
Wharton et al.
Meany et al.
Wugofski et al.
Yen
Logan et al.
Brederveld et al.
Chui
Redford et al.
Hutchinson et al.
Lyons
Abecassis
Tokunaga
Hamilton et al.
Darbee et al.
Yiu
Aharoni et al.
Naimpally
Chui et al.
Heckel
Croy et al.
Driscoll, Jr. et al.
Slivka et al.
FenWick et al.
Logan et al.
Sorber
Abecassis
Allport
Allport
Faroudja et al.
Wang
Appelbaum et al.
Boyce et al.
LinZer et al.
Abecassis
Hayashi et al.
Hendricks et al.
Abecassis
Mershon
Chaddha et al.
Hamilton et al.
Roberts et al.
Spilo et al.
Kikinis
Allport
Margulis
Sampat et al.
Ghori et al.
Ehreth
Margulis et al.
HerZi et al.
Shaw et al.
Kitagawa et al.
Hejna, Jr.
Eyal
Abecassis
Gubbi
Chawla et al.
Margulis
Youn et al.
6,470,378
6,476,826
6,487,319
6,493,874
6,496,122
6,505,169
6,510,177
6,529,506
6,553,147
6,553,178
6,557,031
6,564,004
6,567,984
6,584,201
6,584,559
6,597,375
6,598,159
6,600,838
6,609,253
6,611,530
6,628,716
6,642,939
6,647,015
6,658,019
6,665,751
6,665,813
6,697,356
6,701,380
6,704,678
6,704,847
6,708,231
6,718,551
6,754,266
6,754,439
6,757,851
6,757,906
6,766,376
6,768,775
6,771,828
6,774,912
6,781,601
6,785,700
6,795,638
6,798,838
6,806,909
6,807,308
6,816,194
6,816,858
6,826,242
6,834,123
6,839,079
6,847,468
6,850,571
6,850,649
6,868,083
6,889,385
6,892,359
6,898,583
6,907,602
6,927,685
6,930,661
6,931,451
6,941,575
6,944,880
6,952,595
6,981,050
7,016,337
7,020,892
7,032,000
7,047,305
7,055,166
7,058,376
7,110,558
10/2002
11/2002
11/2002
12/2002
12/2002
1/2003
1/2003
3/2003
4/2003
4/2003
4/2003
5/2003
5/2003
6/2003
6/2003
7/2003
7/2003
7/2003
8/2003
8/2003
9/2003
11/2003
11/2003
12/2003
12/2003
12/2003
2/2004
3/2004
3/2004
3/2004
3/2004
4/2004
6/2004
6/2004
6/2004
6/2004
7/2004
7/2004
8/2004
8/2004
8/2004
8/2004
9/2004
9/2004
10/2004
10/2004
11/2004
11/2004
11/2004
12/2004
1/2005
1/2005
2/2005
2/2005
3/2005
5/2005
5/2005
5/2005
6/2005
8/2005
8/2005
8/2005
9/2005
9/2005
10/2005
12/2005
3/2006
3/2006
4/2006
5/2006
5/2006
6/2006
9/2006
Tracton et al.
Plotkin et al.
Chai
Humpleman
Sampsell
Bhagavath et al.
De Bonet et al.
Yamamoto et al.
Chai et al.
Abecassis
Mimura et al.
Kadono
Allport
Konstantinou et al.
Huh et al.
YaWitZ
McAlister et al.
Chui
SWiX et al.
Apostolopoulos
Tan et al.
Vallone et al.
Malkemes et al.
Chen et al.
Chen et al.
Forsman et al.
Kretschmer et al.
Schneider et al.
Minke et al.
Six et al.
Kitagawa
SWiX et al.
Bahl et al.
Hensley et al.
Park et al.
Look et al.
Price
Wen et al.
Malvar
Ahmed et al.
Cheung
Masud et al.
Skelley, Jr.
Ngo
Radha et al.
Chui et al.
Zhang et al.
Coden et al.
Ojard et al.
Acharya et al.
Barlow et al.
Ferriere
Tardif
Malvar
Apostolopoulos et al.
Rakib et al.
Nason et al.
Rising, III
Tsai et al.
Wathen
Uchida et al.
Logan et al.
Allen
Allen
Ikedo et al.
Tobias et al.
Wu et al.
Levesque et al.
Tripp
Brooks et al.
Logan et al.
Logan et al.
Elliott
US 7,877,776 B2
Page3
7,124,366
7,151,575
7,155,734
7,155,735
7,184,433
7,224,323
7,239,800
7,344,084
7,430,360
7,430,686
7,464,396
7,502,733
7,505,480
7,565,681
7,661,121
2001/0021998
2002/0004839
2002/0010925
2002/0012530
2002/0031333
2002/0046404
2002/0053053
2002/0080753
2002/0090029
2002/0105529
B2
B1
B1
B1
B1
B2
B2
B2
B2
B1
B2
B2
B1
B2
B2
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
10/2006
12/2006
12/2006
12/2006
2/2007
5/2007
7/2007
3/2008
9/2008
9/2008
12/2008
3/2009
3/2009
7/2009
2/2010
9/2001
1/2002
1/2002
1/2002
3/2002
4/2002
5/2002
6/2002
7/2002
8/2002
Foreman eta1.
Landryet a1.
ShiInOmura eta1.
Ngo eta1.
oz
Uchida eta1.
Bilbrey
DaCosta
Abecassis
Wang eta1.
Hejna, Jr.
Andrsen eta1.
Zhang eta1.
Ngo eta1.
Smith eta1.
Margulis
Wine eta1.
Kikinis
Bruls
Mano eta1.
Mizutani
Nagai et a1.
Lee
Kim
Bowser eta1.
2005/0055595
2005/0060759
2005/0097542
2005/0114852
2005/0132351
2005/0138560
2005/0198584
2005/0204046
2005/0216851
2005/0227621
2005/0229118
2005/0246369
2005/0251833
2005/0262539
2005/0283791
2005/0288999
2006/0011371
2006/0015925
2006/0031381
2006/0050970
2006/0051055
2006/0095401
2006/0095472
2006/0095942
2006/0095943
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
3/2005
3/2005
5/2005
5/2005
6/2005
6/2005
9/2005
9/2005
9/2005
10/2005
10/2005
11/2005
11/2005
11/2005
12/2005
12/2005
1/2006
1/2006
2/2006
3/2006
3/2006
5/2006
5/2006
5/2006
5/2006
Frazer eta1.
Rowe eta1.
Lee
Chen eta1.
Randa11eta1.
Lee eta1.
Matthews et a1.
watanabe
Hull eta1.
Katoh
Chiu eta1.
Oreizy eta1.
Schedivy
Barton eta1.
Mocarthyetal.
Lerner et a1.
Fahey
Logan
van Luijt et a1.
Gunatilake
Ohkawa
Krikorian eta1.
Krikorian eta1.
van Beek
Demircin eta1.
2002/0112247 A1
8/2002 Horner et a1.
2006/0107226 A1
2002/0120925
2002/0122137
2002/0131497
2002/0138843
A1
A1
A1
A1
8/2002
9/2002
9/2002
9/2002
Logan
Chen eta1.
Jang
Samaan eta1.
2006/0117371
2006/0146174
2006/0280157
2006/0280437
A1
A1
A1
A1
6/2006
7/2006
12/2006
12/2006
5/2006 Matthews et a1.
Margulis
Hagino
Karaoguz eta1.
Logan eta1.
2002/0143973
2002/0147634
2002/0147687
2002/0167458
2002/0188818
2002/0191575
2003/0001880
2003/0028873
2003/0065915
2003/0093260
2003/0093790
2003/0095791
2003/0115167
2003/0159143
2003/0187657
2003/0192054
2003/0208612
2003/0231621
2004/0003406
2004/0052216
2004/0068334
2004/0083301
2004/0100486
2004/0103340
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
10/2002
10/2002
10/2002
11/2002
12/2002
12/2002
1/2003
2/2003
4/2003
5/2003
5/2003
5/2003
6/2003
8/2003
10/2003
10/2003
11/2003
12/2003
1/2004
3/2004
4/2004
4/2004
5/2004
5/2004
Price
Jacoby eta1.
Breiter et a1.
Baudisch eta1.
Nimura eta1.
Kalavade eta1.
Holtz eta1.
Lemmons
Yu eta1.
Dagtas et a1.
Logan eta1.
Barton eta1.
shanfet a1.
chan
Erhart eta1.
Birks et a1.
Harris et a1.
Gubbi et a1.
Billmaier
Roh
Tsai et a1.
Murase eta1.
Flannni et a1.
Sundareson eta1.
2007/0003224
2007/0005783
2007/0022328
2007/0074115
2007/0076604
2007/0113250
2007/0168543
2007/0180485
2007/0198532
2007/0234213
2007/0286596
2007/0300258
2008/0019276
2008/0036917
2008/0037573
2008/0059533
2008/0134267
2008/0195744
2008/0199150
2008/0294759
2008/0307456
2008/0307462
2008/0307463
2009/0074380
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
1/2007
1/2007
1/2007
3/2007
4/2007
5/2007
7/2007
8/2007
8/2007
10/2007
12/2007
12/2007
1/2008
2/2008
2/2008
3/2008
6/2008
8/2008
8/2008
11/2008
12/2008
12/2008
12/2008
3/2009
Krikorian eta1.
Saint-Hillaire eta1.
Tarra eta1.
Patten eta1.
Litwack
Logan eta1.
Krikorian eta1.
Dna
Krikorian eta1.
Krikorian eta1.
Lonn
O’Connoretal.
Takatsuji eta1.
Pascarella eta1.
Cohen
Krikorian
Moghe eta1.
Bowraetal.
Candelore
Biswas et a1.
Beetcher et a1.
Beetcher et a1.
Beetcher et a1.
Boston eta1.
2004/0139047 A1
7/2004 Rechsteiner et a1.
2009/0199248 Al*
8/2009 Ngo eta1. ................... .. 725/90
2004/0162845 A1
2004/0162903 A1
8/2004 Kim eta1.
8/2004 Oh
2010/0100915 A1
4/2010 Krikorian eta1.
2004/0172410 A1
2004/0205830
2004/0212640
2004/0216173
2004/0236844
2004/0255249
2004/0255330
2004/0255334
2004/0255336
2005/0005308
2005/0021398
2005/0027821
2005/0038981
2005/0044058
2005/0050462
2005/0053356
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
9/2004 Shimojima et a1.
10/2004
10/2004
10/2004
11/2004
12/2004
12/2004
12/2004
12/2004
1/2005
1/2005
2/2005
2/2005
2/2005
3/2005
3/2005
Kaneko
Mann eta1.
Horoszowski et a1.
Kocherlakota
chang eta1.
Logan
Logan
Logan eta1.
Logan eta1.
McCleskey eta1.
Alexander et a1.
Connor etal.
Matthews et a1.
Whittle eta1.
Mate et a1.
FOREIGN PATENT DOCUMENTS
DE
EP
EP
EP
EP
EP
GB
JP
JP
JP
KR
KR
WO
W0
4407319
0838945
1077407
1443766
1691550
1830558
2307151
2003046582
2003114845
2004015111
19990082855
20010211410
0133839
WO 01/33839
A1
A
A1
A2
A
A1
A
A
A
A
A
A
A1
9/1994
4/1998
2/2001
8/2004
8/2006
9/2007
5/1997
2/2003
4/2003
V2004
ll/l999
8/2001
5/2001
5/2001
US 7,877,776 B2
Page 4
WO
WO
WO
WO
W0
WO
WO
WO
WO
WO
WO
WO
WO
WO
WO
0147248
0193161
03026232
03052552
W0 03/052552
03098897
2004032511
2005050898
2006064454
2006074110
2007027891
2007051156
2007141555
2007149466
2008024723
A2
A1
A1
A2
A
A1
A2
A1
A
A2
A2
A2
A2
A
6/2001
12/2001
3/2003
6/2003
6/2003
11/2003
4/2004
6/2005
6/2006
7/2006
3/2007
5/2007
12/2007
12/2007
2/2008
OTHER PUBLICATIONS
International Search Report for International Application No. PCT/
US2007/076337 mailed Oct. 20. 2008.
Rao, PadmanabhaR.,U.S.Appl.No. 12/347,465, ?led Dec. 31, 2008.
European Patent Of?ce, European Search Report for European
Application No. EP 08 16 7880, mailed Mar. 4, 2009.
MythTV Wiki, “MythTV User Manual” [Online], Aug. 27, 2007,
XP002515046; retrieved from the Internet: <URL: http://www.
mythtv.org/wiki?title:UseriManual:Introduction&oldid:25549>.
International Searching Authority, Written Opinion and International
Search Report for International Application No. PCT/US2008/
077733, mailed Mar. 18, 2009.
International Searching Authority, Written Opinion and International
Search Report for International Application No. PCT/US2008/
087005, mailed Mar. 20, 2009.
Watanabe Y. et al., “Multimedia Database System for TV Newscasts
and Newspapers”; Lecture Notes in Computer Science, Springer
Verlag, Berlin, Germany; vol. 1554, Nov. 1, 1998, pp. 208-220,
myreplaytv.com/ Aug. 15, 2000, entire document.*
XP002402824, ISSN: 0302-9743.
Yasuhiko Watanabe et al., “Aligning Articles in TV Newscasts and
Newspapers”; Proceedings of the International Conference on
Faucon, B., “TV ‘Brick’ Opens up Copyright Can of Worms,” Finan
cial Review, Jul. 1, 2003, [online] [Retrieved on Apr. 12, 2006]
XP002402825.
http://web.archive.org/web/20000815052751/http://www.
Retrieved
from
the
Internet
<URL:http://afr.com/cgi-bin/
newteXtversions.pl?storyid:1056825330084&date:2003/07/01&
pagetype:printer&section:1053 801318705&path:/aIticles/2003/
06/30/1056825330084.html >.
Computationallinguistics, XX, XX, Jan. 1, 1998, pp. 1381-1387,
Sodergard C. et al., “Integrated Multimedia Publishing: Combining
TV and Newspaper Content on Personal Channels”; Computer Net
works, Elsevier Science Publishers B.V., Amsterdam, Netherlands;
vol. 31, No. 11-16, May 17, 1999, pp. 1111-1128, XP004304543,
ISSN: 1389-1286.
Ariki Y. et al., “Automatic Classi?cation of TV News Articles Based
Archive of “TV Brick Home Server,” www.tvbrick.com, [online]
[Archived by http://archive.org on Jun. 3, 2004; Retrieved on Apr. 12,
2006] Retrieved from the Internet <URL:http://web.archive.org/
on Telop Character Recognition”; Multimedia Computing and Sys
web/20041 10711 1024/www.tvbrick.com/en/af?liate/tvbs/tvbrick/
tems, 1999; IEEE International Conference on Florence, Italy, Jun.
tvbrick/document18iprint >.
PCT International Search Report and Written Opinion, PCT/US05/
20105, Feb. 15, 2007, 6 pages.
7-11, 1999, Los Alamitos, CA, USA, IEEE Comput. Soc. US; vol. 2,
Jun. 7, 1999, pp. 148-152, XP010519373, ISBN: 978-0-7695-0253
Krikorian, Jason, U.S. Appl. No. 11/734,277, ?led Apr. 12, 2007.
Tarra, Raghuveer et al., U.S. Appl. No. 60/975,239, ?led Sep. 26,
2007.
Rao, Padmanabha R., U.S. Appl. No. 12/166,039, ?led Jul. 1,2008.
Williams, George Edward, U.S. Appl. No. 12/167,041, ?led Jul. 2,
2008.
Faucon, B. “TV ‘Brick’ Opens up Copyright Can of Worms, ”Finan
cial Review, Jul. 1, 2003, [online [Retrieved on Apr. 12, 2006]
Retrieved from the Internet, URL:http://afr.com/c gi-bin/newtextver
sions.pl?storyid+1056825330084&3ate+2003/07/01&
pagetype+printer&section+1053 801318705&path+articles/2003/
06/30/0156825330084.html.].
Archive of “TV Brick Home Server,” www.tvbrick.com, [online]
[Archived by http://archive.org on Jun. 3, 2004; Retrieved on Apr. 12,
2006] retrieved from the Internet <URL:http://web.archive.org/web/
200411071 11024/www.tvbrick.com/en/af?liate/tvbs/tvbriclddocu
ment18/print>.
PCT International Search Report and Written Opinion, PCT/
US2005/020105, Feb. 15, 2007, 6 pages.
Roe, Kevin, “Third-Party Observation Under EPC Article 115 On
The Patentability Of An Invention,” Dec. 21, 2007.
Roe, Kevin, Third-Party Submission For Published Application
Under CFR §1.99, Mar. 26, 2008.
Skodras et al., “JPEG2000: The Upcoming Still Image Compression
Standard,” May 11, 2000, 14 pages.
Kessler, Gary C., An Overview of TCP/IP Protocols and the Internet;
Jan. 16, 2007, retrieved from the Internet on Jun. 12, 2008 at http://
3; abstract, paragraph [03.1], paragraph [05.2], ?gures 1,2.
USPTO, Non-Final Of?ce Action mailed Dec. 17, 2004; U.S. Appl.
No. 09/809,868, ?led Mar. 15, 2001.
USPTO, Final Of?ce Action mailed Jul. 28, 2005; U.S. Appl. No.
09/809,868, ?led Mar. 15, 2001.
USPTO, Non-Final Of?ce Action mailed Jan. 30, 2006; U.S. Appl.
No. 09/809,868, ?led Mar. 15, 2001.
USPTO, Final Of?ce Action mailed Aug. 10, 2006; U.S. Appl. No.
09/809,868, ?led Mar. 15, 2001.
USPTO, Non-Final Of?ce Action mailed Jun. 19, 2007; U.S. Appl.
No. 09/809,868, ?led Mar. 15, 2001.
USPTO, Non-Final Of?ce Action mailed Apr. 16, 2008; U.S. Appl.
No. 09/809,868, ?led Mar. 15, 2001.
USPTO, Final Of?ce Action mailed Sep. 18, 2008; U.S. Appl. No.
09/809,868, ?led Mar. 15, 2001.
USPTO, Non-Final Of?ce Action mailed Mar. 31, 2009; U.S. Appl.
No. 09/809,868, ?led Mar. 15, 2001.
USPTO, Non-Final Of?ce Action mailed May 1, 2008; U.S. Appl.
No. 11/111,265, ?led April 21, 2005.
USPTO, Final Of?ce Action mailed Dec. 29, 2008; U.S. Appl. No.
11/111,265, ?led Apr. 21, 2005.
USPTO, Non-Final Of?ceAction mailed Jun. 8, 2009; U.S.Appl. No.
11/111,265, ?led Apr. 21, 2005.
USPTO, Non-Final Of?ce Action mailed Jun. 26, 2008; U.S. Appl.
No. 11/620,707, ?led Jan. 7, 2007.
USPTO, Final Of?ce Action mailed Oct. 21, 2008; U.S. Appl. No.
11/620,707, ?led Jan. 7, 2007.
USPTO, Non-Final Of?ce Action mailed Mar. 25, 2009; U.S. Appl.
No. 11/620,707, ?led Jan. 7, 2007.
www.garykesslernet/library/tcpip.htrnl; originally submitted to the
USPTO, Non-Final Of?ce Action mailed Aug. 7, 2008; U.S. Appl.
InterNIC and posted on their Gopher site on Aug. 5, 1994.
Taubman et al., “Embedded Block Coding In JPEG2000,” Feb. 23,
2001, pp. 1-8 of36.
Balster, Eric J ., “Video Compression and Rate Control Methods
Based on the Wavelet Transform,” The Ohio State University 2004,
No. 11/620,711, ?led Jan. 7, 2007.
USPTO, Final Of?ce Action mailed Feb. 9, 2009; U.S. Appl. No.
11/620,711, ?led Jan. 7, 2007.
USPTO, Non-Final Of?ce Action mailed Feb. 25, 2009; U.S. Appl.
No. 11/683,862, ?led Mar. 8,2007.
USPTO, Non-Final Of?ce Action mailed Dec. 24, 2008; U.S. Appl.
No. 11/147,985, ?led Jun. 7, 2005.
USPTO, Non-Final Of?ce Action mailed Jun. 25, 2008; U.S. Appl.
No. 11/428,254, ?led Jun. 30, 2006.
USPTO, Final Of?ce Action mailed Feb. 6, 2009; U.S. Appl. No.
11/428,254, ?led Jun. 30, 2006.
pp. 1-24.
Kulapala et al., “Comparison of Traf?c and Quality Characteristics of
Rate-Controlled Wavelet and DCT Video,” AriZona State University,
Oct. 11, 2004.
International Search Report and Written Opinion for PCT/US2006/
04382, mailed Apr. 27, 2007.
US 7,877,776 B2
Page 5
Veri?cation & Quali?cation: Bluetooth Technical Background, Apr.
21, 1999; pp. 1 of7 and 2 of 7; http://www.bluetooth.com/v2/news/
show.asp 1-2.
International Search Report for PCT/US2008/069914 mailed Dec.
19, 2008.
PCT Partial International Search, PCT/U S2009/ 054893, mailed Dec.
23, 2009.
Microsoft Corporation; Harman/Kardon “Master Your Universe”
European Patent Of?ce, European Search Report, mailed Sep. 28,
1999.
Matsushita Electric Corporation of America MicroCast : Wireless PC
2009 for European Application No. EP 06 78 6175.
DitZe M. et all “Resource Adaptation for Audio -Visual Devices in the
Multimedia Transceiver System, Nov. 1998.
“Wireless Local Area Networks: Issues in Technology and Stan
dards” Jan. 6, 1999.
USPTO, Final Of?ce Action mailed Jun. 25, 2009; U.S. Appl. No.
11/147,985, ?led Jun. 7, 2005.
China State Intellectual Property Of?ce “First Of?ce Action,” issued
UPnP QoS Architecture,” Advanced Networking and Applications,
Jul. 31, 2009, for Application No. 200580026825.X.
USPTO, Non-Final Of?ce Action, mailed Aug. 4, 2009; U.S. Appl.
No. 11/734,277, ?led Apr. 12, 2007.
USPTO, Final Of?ce Action, mailed Jul. 31, 2009; U.S. Appl. No.
11/683,862, ?led Mar. 8, 2007.
USPTO, Non-Final Of?ce Action, mailed Aug. 5, 2009; U.S. Appl.
No. 11/147,663, ?led Jun. 7, 2005.
USPTO, Non-Final Of?ce Action, mailed Sep. 3, 2009; U.S. Appl.
No. 11/620,711, ?led Jan. 7,2007.
Einaudi, Andrew E. et al. “Systems and Methods for Selecting Media
Content Obtained from Multiple Sources,” U.S. Appl. No.
12/543,278, ?led Aug. 18, 2009.
Malode, Deepak Ravi “Remote Control and Method for Automati
cally Adjusting the Volume Output of an Audio Device,” U.S. Appl.
No. 12/550,145, ?led Aug. 28, 2009.
Akella, Aparna Sarma “Systems and Methods for Event Program
ming Via a Remote Media Player,” U.S. Appl. No. 12/537,057, ?led
Aug. 6, 2009.
Shah, Bhupendra Natwerlan et al. “Systems and Methods for
Transcoding and Place Shifting Media Content,” U.S. Appl. No.
USA, Jan. 23-27, 2006.
Lowekamp, B. et al. “A Hierarchy of Network Performance Charac
teristics for Grid Applications and Services,” GGF Network Mea
12/548,130, ?led Aug. 26, 2009.
16, 2009.
Rao, Padmanabha R. et al. “Methods and Apparatus for Establishing
Network Connections Using an Inter-Mediating Device,” U.S. Appl.
No. 12/642,368, ?led Dec. 18, 2009.
Dham, Vikram et al. “Systems and Methods for Establishing Network
Connections Using Local Mediation Services,” U.S. Appl. No.
12/644,918, ?led Dec. 22, 2009.
Paul, John et al. “Systems and Methods for Remotely Controlling
Media Server Via a Network,” U.S. Appl. No. 12/645,870, ?led Dec.
23, 2009.
Bajpal, Parimal et al. “Method and Node for Transmitting Data Over
a Communication Network using Negative Ackhowledgement,” U.S.
Appl. No. 12/404,920, ?led Mar. 16, 2009.
Bajpal, Parimal et al. “Method and Note for Employing Network
connections Over a Connectinoless Transport Layer Protocol,” U.S.
Appl. No. 12/405,062, ?led Mar. 16, 2009.
Asnis, Ilya et al. “Mediated Network address Translation Traversal”
U.S. Appl. No. 12/405,039, ?led Mar. 16,2009.
Australian Government “Of?ce Action,” Australian Patent Applica
tion No. 2006240518, mailed Nov. 12, 2009.
Jain, Vikal Kumar “Systems and Methods for Coordinating Data
Communication Between Two Device,” U.S. Appl. No. 12/ 699,280,
?led Feb. 3, 2010.
Gangotri, Arun L. et al. “Systems and Methods and Program Appli
Bluetooth-News;
Main
Future
User
Models
Document
Banger, Shashidhar et al. “Systems and Methods for Automatically
Controlling the Resolution of Streaming Video Content,” U.S. Appl.
No. 12/537,785, ?led Aug. 7, 2009.
Panigrahi, Biswaranjan “Home Media Aggregator System and
Method,” U.S. Appl. No. 12/538,681, ?led Aug. 10,2009.
Nandury, Venkata Ki shore “Adaptive Gain Control for Digital Audio
Samples in a Media Stream,” U.S. Appl. No. 12/507,97 1, ?led Jul. 23,
2009.
Shirali, Amey “Systems and Methods for Providing Programming
Content,” U.S. Appl. No. 12/538,676, ?led Aug. 10,2009.
Thiyagaraj an, Venkatesan “Systems and Methods for Virtual Remote
Control of Streamed Media,” U.S. Appl. No. 12/538,664, ?led Aug.
10, 2009.
Thiyagarajan, Venkatesan et al. “Localization Systems and Method,”
U.S. Appl. No. 12/538,783, ?led Aug. 10, 2009.
Shirali, Amey et al. “Methods and Apparatus for Seeking Within a
Media Stream Using Scene Detection,” U.S. Appl. No. 12/ 538,784,
?led Aug. 10, 2009.
Thiyagarajan, Venkatesan “Systems and Methods for Updating
Firmware Over a Network,” U.S. Appl. No. 12/538,661, ?led Aug.
10, 2009.
Iyer, Sati sh “Methods and Apparatus for Fast Seeking Within a Media
Stream Buffer,” U.S. Appl. No. 12/538,659, ?led Aug. 10,2009.
European Patent Of?ce, International Searching Authority, “Interna
tional Search Report,” for International Application No. PCT/
US2009/049006, mailed Sep. 11, 2009.
China State Intellectual Property Of?ce “First Of?ce Action,” issued
Jan. 8, 2010, for Application No. 200810126554.0.
USPTO, Final Of?ce Action mailed Nov. 6, 2009; U.S. Appl. No.
09/809,868, ?led Mar. 15, 2001.
USPTO, Final Of?ce Action mailed Nov. 12, 2009; U.S. Appl. No.
11/620,707, ?led Jan. 7, 2007.
USPTO, Non-Final Of?ce Action mailed Oct. 1, 2009; U.S. Appl.
No. 11/778,287, ?led Jul. 16,2007.
USPTO, Non-Final Of?ce Action mailed Nov. 23, 2009, U.S. Appl.
No. 11/683,862, ?led Mar. 8, 2007.
USPTO Final Of?ce action mailed Jan. 25, 2010; U.S. Appl. No.
11/734,277, ?led Apr. 12,2007.
2006; AINA, 2006; 20% H International conference on Vienna, Aus
tria Apr. 18-20, 2006.
Joonbok, Lee et al. “Compressed High De?nition Television
(HDTV) Over IPv6,” Applications and the Internet Workshops, 2006;
Saint Workshops, 2006; International Symposium, Phoenix, AZ,
surements Working Group, pp. 1-29, May 24, 2004.
Newton’s Telcom Dictionary, 21st ed., Mar. 2005.
Sling Media “Sling Media Unveils Top-of-Line SlingboX PRO-HD”
[online], Jan. 4, 2008, XP002560049; retrieved from the Internet:
URL:www.slingmedia.com/get/pr-slingboX-pro-hd.html; retrieved
on Oct. 12, 2009.
Srisuresh, P. et al. “Traditional IP Network Address Translator (Tra
ditional NAT),” Network Working Group, The Internet Society, Jan.
2001.
Lucas, Brian et al. “Systems and Methods for Establishing Connec
tions Between Devices Communicating Over aNetwork,” U.S. Appl.
No. 12/426,103, ?led Apr. 17,2009.
Thiyagaraj an, Venkatesan et al. “Always-On-Top Media Player
Launched From a Web Browser,” U.S. Appl. No. 12/617,271, ?led
Nov. 12, 2009.
Paul, John Michael et al. “Systems and Methods for Delivering
Messages Over a Network,” U.S. Appl. No. 12/619,192, ?led Nov.
cations for Selectively Restructuring the Placeshiftnig of Copy Pro
tected Digital Media Content,” U.S. Appl. No. 12/ 623,955, ?led Nov.
23, 2009.
Paul, John et al. “Systems and Methods for Searching Media Con
tent,” U.S. Appl. No. 12/648,024, ?led Dec. 28, 2009.
Carlson, T. “Mule 2.x Getting Started Guide,” Apr. 15, 2008, 134
pages.
Curtis, Sean “Systems and Methods for Formatting Media Content
for Distribution,” U.S. Appl. No. 12/635,138, ?led Dec. 10, 2009.
Newton’s Telcom Dictionary, 20th ed., Mar. 2004.
“The Authoritative Dictionary of IEEE Standard Terms,” 7th ed.
2000.
GurZhi, Alexander et al. “Systems and Methods for Emulating Net
work-Enabled Media Components,” U.S . Appl. No. 12/711,830, ?led
Feb. 24, 2010.
US 7,877,776 B2
Page 6
Bajpai, Parimal et al. “Systems and Methods of Controlling the
Encoding ofa Media Stream,” U.S. Appl. No. 12/339,878, ?led Dec.
19, 2008.
Malone, Edward D. et al. “Systems and Methods for Controlling
Media Devices,” U.S. Appl. No. 12/256,344, ?led Oct. 22, 2008.
Banger, Shashidhar et al. “Systems and Methods for Determining
Attributes of Media Items Accessed Via a Personal Media Broad
caster,” U.S. Appl. No. 12/334,959, ?led Dec. 15,2008.
Kulkarni, Anant Madhava “Systems and Methods for Creating Logi
cal Media Streams for Media Storage and Playback,” U.S. Appl. No.
12/323,907, ?led Nov. 26, 2008.
Rao, Padmanabha R. “Systems and Methods for Linking Media
Content,” US. Appl. No. 12/359,784, ?led Jan. 26, 2009.
Krikorian, Blake Gary et al. “Systems and Methods for Presenting
Media Content Obtained From Multiple Sources,” US. Appl. No.
12/408,456, ?led Mar. 20, 2009.
Krikorian, Blake Gary et al. “Systems and Methods for Projecting
Images From a Computer System,” US. Appl. No. 12/408,460, ?led
Mar. 20, 2009.
International Search Report and Written Opinion for International
Application No. PCT/US2006/025911, mailed Jan. 3, 2007.
International Search Report for International Application No. PCT/
US2007/063599, mailed Dec. 12, 2007.
International Search Report and Written Opinion for International
Application No. PCT/US2008/080910, mailed Feb. 16, 2009.
International Search Report and Written Opinion for International
Application No. PCT/US2006/025912, mailed Jul. 17, 2008.
International Search Report for International Application No. PCT/
US2008/059613, mailed Jul. 21, 2008.
Wikipedia “LocationFree Player” [Online], Sep. 22, 2007,
XP002512400; retrieved from the Internet: <URL: http://en.
wikipedia.org/w/index.php?title:LocationFreeiPlayer&
USPTO Final Of?ce Action mailed Mar. 12, 2010; US. Appl. No.
11/620,711, ?led Jan. 7, 2007.
USPTO Final Of?ce Action mailed Mar. 3, 2010; US. Appl. No.
11/111,265, ?led Apr. 21, 2005.
USPTO Non-Final Of?ce Action mailed Mar. 31, 2010; US. Appl.
No. 11/620,707, ?led Jan. 7, 2007.
USPTO Non-Final Of?ce Action mailed Apr. 1, 2010; US. Appl. No.
12/237,103, ?led Sep. 24, 2008.
Qiong, Liu et al. “Digital Rights Management for Content Distribu
tion,” Proceedings of the Australasian Information Security Work
shop Conference on ACSW Frontiers 2003, vol. 21, 2003,
XP002571073, Adelaide, Australia, ISSN: 1445-1336, ISBN:
1-920682-00-7, sections 2 and 2.1.1.
China State Intellectual Property Of?ce “Of?ce Action” issued Mar.
18, 2010 for Application No. 2006800225206.
Canadian Intellectual Property Of?ce “Of?ce Action” mailed Feb.
18, 2010 for Application No. 2569610.
European Patent Of?ce “European Search Report,” mailed May 7,
2010 for Application No. 067861740.
China State Intellectual Property Of?ce “Of?ce Action” issued Apr.
13, 2010 for Application No. 200580026825.X.
Margulis, Neal “Apparatus and Method for Effectively Implementing
aWireless Television System,” US. Appl. No. 12/758,193, ?led Apr.
12, 2010.
Margulis, Neal “Apparatus and Method for Effectively Implementing
aWireless Television System,” US. Appl. No. 12/758,194, ?led Apr.
12, 2010.
Margulis, Neal “Apparatus and Method for Effectively Implementing
aWireless Television System,” US. Appl. No. 12/758,196, ?led Apr.
12, 2010.
Kirkorian, Jason Gary et al. “Personal Media Broadcasting System
with Output Buffer,” U.S. Appl. No. 12/757,697, ?led Apr. 9, 2010.
Tarra, Raghuveer et al. “Firmware Update for Consumer Electronic
oldid:159683564>; retrieved on Jan. 28, 2009.
Device,” U.S. Appl. No. 12/757,714, ?led Apr. 9, 2010.
Wikipedia “Slingbox” [Online], Oct. 21, 2007, XP002512399;
Lee, M. et al. “Video Frame Rate Control for Non-Guaranteed Net
retrieved from the Internet: <URL:http://en.wikipedia.org/w/index.
php?title:Slingbox&oldid:166080570>; retrieved on Jan. 28, 2009.
Capable Networks LLC “Keyspan Remote Control4Controlling
work Services with Explicit Rate Feedback,” Globecom’00, 2000
IEEE Global Telecommunications conference, San Francisco, CA,
Nov. 27-Dec. 1, 2000; [IEEE Global Telecommunications Confer
Your Computer With a Remote” [Online], Feb. 21, 2006,
XP002512495; retrieved from the Internet: <URL:http://www.
XP001195580; ISBN: 978-0-7803-6452-3, lines 15-20 of sec. II on
slingcommunity.com/article/1 1791/Keyspan-Remote-Control-Controlling-Your-Computer-With-a-Remote/?
highlight:remote+control>; retrieved on Jan. 28, 2009.
Sony Corporation “LocationFree Player PakiLocationFree Base
StationiLocationFree Player” [Online] 2005, XP002512401;
retrieved from the Internet: <URL:http://www.docs.sony.com/re
lease/LFPK1.pdf>; retrieved on Jan. 28, 2009.
Sling Media Inc. “Slingbox User Guide” [Online] 2006,
XP002512553; retrieved from the Internet: <URL:http://www.
ence], NewYork, NY; IEEE, US, vol. 1,Nov. 27, 2000, pp. 293-297,
p. 293, ?g. 1.
European Patent Of?ce, International Searching Authority, “Interna
tional Search Report and Written Opinion,” mailed Jun. 4, 2010 for
International Application No. PCT/IN2009/000728, ?led Dec. 18,
2009.
USPTO Non-Final Of?ce Action mailed Jun. 23, 2010; US. Appl.
No. 11/933,969, ?led Nov. 1,2007.
Korean Intellectual Property Of?ce “Of?cial Notice of Preliminary
Rejection,” issued Jun. 18, 2010; Korean Patent Application No.
slingmediahIdattach/en-USiSlingboxiUseriGuideiv1 .2 .pdf>;
10-2008-7021254.
retrieved on Jan. 29, 2009.
Japan Patent Of?ce “Notice of Grounds for Rejection (Of?ce
Action),” mailed May 25, 2010; Patent Application No. 2007
Sony
Corporation
“LocationFree
TV”
[Online],
2004,
SP002512410; retrieved from the Internet: <URL:http://www.docs.
sony.com/releaselLFX1iX5revision.pdf>; retrieved on Jan. 28,
2009 [noteidocument uploaded in two parts as ?le exceeds the
25MB siZe limit].
European Patent Of?ce, International Searching Authority, “Interna
tional Search Report,” mailed Mar. 30, 2010; International Applica
tion PCT/US2009/068468 ?led Dec. 27, 2009.
0268269.
Japan Patent Of?ce “Notice of Grounds for Rejection (Of?ce
Action),” mailed May 25, 2010; Patent Application No. 2007
527683.
USPTO Non-Final Of?ce Action mailed Aug. 11, 2010; US. Appl.
No. 11/111,265 ?led Apr. 21, 2005.
* cited by examiner
US. Patent
Jan. 25, 2011
Sheet 1 012
US 7,877,776 B2
Central
Server
180
H
e‘m1mwm1acmy
m
06
6 t.
B
Pm68rdnl0&0 mm
n
sO s
r
MDm5 we
as
0
SNb6%1Wer2vwM3 bo
er
Local Clients
150
Remote Clients
170
FIG. 1
Home Internet
Gateway
1_1Q
E
Q:
IR Emitter
E
FIG. 2
US. Patent
Jan. 25, 2011
US 7,877,776 B2
Sheet 2 0f 2
6:250 motE
I26xmo83tV06Bi2E
%a
.OEm
.82>aqwlmmIE
aB29E:;2m“El?m|o
motuE
5o9wAnl_:>.w
mmm
8m
Al
25%: 2 %
US 7,877,776 B2
1
2
PERSONAL MEDIA BROADCASTING
SYSTEM
users to access their A/V content from any of a variety of
remote locations inside or outside the home, as such content
is currently available only from locations in the home that
traditionally receive and play it (e.g., a television set).
CROSS REFERENCE TO RELATED
APPLICATIONS
SUMMARY OF THE INVENTION
This application claims the bene?t of US. Provisional
A personal media broadcasting system enables video trans
Application No. 60/577,833, ?led Jun. 7, 2004, Which is
incorporated by reference in its entirety. This application is
also related to co-pending US. Application entitled, “Fast
Start Streaming and Buffering of Streaming Content for Per
sonal Media Player,” to Krikorian et al., ?led Jun. 7, 2005, and
co-pending US. Application entitled, “Personal Media
Broadcasting System With Output Buffer,” to Krikorian et al.,
?led Jun. 7, 2005, each of Which is incorporated by reference
in its entirety.
mission over a computer netWork, alloWing a user to vieW and
control media sources over a computer netWork from a
remote location. In one embodiment, the personal media
broadcasting system includes a personal media broadcaster
and a media player. The personal media broadcaster may be
con?gured to receive as an input virtually any media source.
The broadcaster digitiZes and compresses the received media
content (if necessary) and streams the media over a computer
netWork to the media player. The media player may reside on
any of a Wide range of client devices for vieWing the media. A
BACKGROUND
1. Field of the Invention
20
This invention relates generally to personal streaming
media broadcasters, and in particular to streaming media
by the media player.
In one embodiment, a personal media broadcasting system
includes a personal media broadcaster and media player cli
from a media source input to a client device over a network.
2. Background of the Invention
While people spend a great deal of time Watching televi
sion programming and other forms of audio-visual (A/V)
content, they are also spending an increasing amount of time
interfacing With computing devices such as personal comput
25
ent, Which communicate With each other over a netWork. The
personal media broadcaster can accept an A/V signal from
one or more A/V source devices. From this A/V signal, the
ers, personal digital assistants, mobile phones, dedicated
multimedia devices, and other devices that, like the tradi
tional television, include a display. These types of computing
user may send control commands (e.g., “channel up”) using
the media player back over the netWork to be executed by the
media source device, thus affecting the media stream received
30
personal media broadcaster constructs a media stream that is
suitable for transmission over a netWork to the media player
client. The personal media broadcaster includes a netWork
interface for transmitting the media stream to the media
devices alloW people to be increasingly mobile, but this
mobility reduces the time people spend at home in front of
player client. As the media stream is received at the client, it
their televisions. It Would therefore be bene?cial to enable
of A/V content they noW receive at home on these computing
embodiment, the personal media broadcaster is a dedicated
appliance, not a general purpose computer. In this Way, a
general purpose computer need not be poWered on and con
devices as Well, regardless of location and Without depen
nected to theA/V source devices for the user to receive media
dence on physical connections.
content remotely. In another embodiment, the media player
people to enjoy their television programming and other forms
can be vieWed by a user using the media player client. In one
35
This ability Would enable several desirable applications.
For example, a user might Want to access and control televi
sion and other regularly consumed A/V content from a per
sonal computer (desktop as Well as notebook computers) or
other computing devices around the home via the user’ s local
netWork in the home. Since cable, satellite, and other sources
of television content typically enter the house at a feW discrete
client can run on any general or multi-purpose device (such as
40
a personal computer or cellular phone), bene?cially avoiding
the need for the user to carry special equipment to use the
45
locations, alloWing access to the content over a home netWork
gives the user more freedom to enjoy the content throughout
the home. Another possible application Would be to enable a
broadcasting system. The computer netWork over Which the
broadcaster and media player client communicate may com
prise a Wide area netWork, such as the Internet, alloWing the
user to receive media content from the home to anyWhere in
the World Where a connection to the netWork is available.
To improve the user’ s experience, the person media broad
casting system may employ any of a number of techniques for
user to access and control television and other A/V content
buffering, transmitting, and vieWing the content. In one
from any number of remote netWorks Where a broadband 50 embodiment, for example, the media stream is constructed by
connection is available to the user (e.g., at an airport, at Work,
at school, in a hotel, in a cafe, at an acquaintance’s house).Yet
encoding the audio/visual signal using a bit rate selected at
another application Would be to enable a user to access and
intermediate output buffer used to temporarily store the
media stream before it is transmitted to the media player
least in part based on an amount of free space remaining in an
control television and otherA/V content from a mobile phone
or other computing devices that can be connected to a Wide
55
area netWork (e.g., GPRS, W-CDMA, CDMA-2000, lXRTT,
lXEVDO, and the like). In various applications, users are
likely to Want to access their media content stored on devices,
such as personal computers and other devices having storage,
from remote netWorks. Nevertheless, netWork bandWidth and
other limitations have made it dif?cult to provide an effective
and enjoyable remote media experience for the user.
But traditional streaming media solutions do not enable
these applications in any effective Way; moreover, they suffer
from technical limitations that Would prevent their use in
personal media broadcasting applications like those
described above. Accordingly, it Would be desirable to enable
60
client. While various measures of occupancy of the interme
diate output buffer can be used, one involves the use of mul
tiple Watermarks to measure the free space available in the
buffer. The encoding bit rate can be decreased each time the
free space dips beloW a Watermark, While it can be increased
When the free space rises above a Watermark or if the free
space never drops beloW the Watermark in the ?rst place. In
this Way, the encoding bit rate can be dynamically adjusted
based on current performance of the system, and this dynamic
65
adjustment can occur at the broadcaster Without requiring
feedback from the client.
In another embodiment, the media player client imple
ments a fast-start mechanism by Which the media player
US 7,877,776 B2
4
3
The system described herein may be applied in a number of
receives a media stream in real time from the personal media
broadcaster and stores the media stream in a buffer. The
applications or usage scenarios. For example, there are a
variety of uses in the home over a Wired or Wireless home
media player client plays the received media stream from the
buffer at a decreased playback speed Without Waiting for the
buffer to ?ll to a predetermined threshold. While the playback
speed is kept at a sloWer rate than the streaming media content
netWork for the system. In the home, users can vieW and
control their A/V source components, such as a personal
video recorder (PVR) or a cable set-top box, from any desktop
PC, notebook PC, PDA, or other netWork devices. For
is received, the buffer ?lls (albeit at a rate sloWer than the
media stream is received). This alloWs the media player client
to play a received media stream Without Waiting for its buffer
example, this application alloWs a user to convert a Wireless
notebook PC situated in the kitchen into a Wireless LCD TV.
As another example of in-home use, young parents can pair a
to ?lls, Which improves the user’s experience dramatically in
personal broadcaster With a simple analog camera and place
situations such as Where the user is changing channels or
operating a menu on the remote A/V source device. In addi
them in the baby’ s room to provide a real time video and audio
stream of their child displayed on a device in another location
in the home.
Another usage scenario category relates to users Who are
aWay from home but have access to a broadband connection.
This access can be in a variety of locations, such as at Work,
at school, at a friend’s house, in a cafe, at the airport or in a
tion, the media player client may further enhance this expe
rience by sWitching into a control mode When the client
receives a user command to control content in the media
stream. When the client enters the control mode, the buffer is
?ushed and the received media stream is played. In this Way,
the content in the buffer need not be displayed on the client
device, and the user can then almost immediately receive
visual feedback con?rming the user’s command to control
plane, or in a hotel. From these locations, users can connect to
20
content in the media stream.
their personal broadcaster over the public Internet and enjoy
the same live television experience that they have at home.
The present invention alloWs users that are aWay from home
to access the full breadth of programming they are accus
BRIEF DESCRIPTION OF THE DRAWINGS
tomed to, and even vieW content that they previously captured
FIG. 1 is a block diagram of a media broadcasting system,
in accordance With an embodiment of the invention.
FIG. 2 is a block diagram of a set of input and output
connections for a personal media broadcaster, in accordance
25
With an embodiment of the invention.
FIG. 3 is a block diagram of a personal media broadcaster,
in accordance With an embodiment of the invention.
30
DETAILED DESCRIPTION OF THE PREFERRED
EMBODIMENTS
35
OvervieW
Embodiments of the invention alloW for distribution ofA/V
RTSP, RTP, RSVP, and the like. Because embodiments of the
invention can accept, digitiZe, encode, and stream any analog
content from a variety of A/V source devices over a computer
netWork to a client device. As used herein, A/V content may
on their personal video recorder.
Yet another usage scenario category relates to individuals
With mobile phones, communicators, or other Wide area net
Work devices. On emerging netWorks, the bit rate is noW high
enough for users to enjoy streaming video on their devices.
This opens up neW possibilities for the personal broadcaster
to deliver a Wealth of programming to the mobile user, all of
Which originates from the user’s oWn home.
The media stream transmitted from the personal media
broadcaster to the media player client netWorks can take
advantage of a combination of any number of netWorking
protocols, including HTTP over TCP/IP, as Well as UDP,
40
A/V source, there are a vast number of uses for the personal
broadcasterifrom a security camera system to a method for
include any type of media content, including audio or video
extending a user’s satellite radio to the user’s cell phone.
content, or both. In one embodiment, a personal media broad
caster takes as an input an audio or video signal, digitiZes and
System Architecture
FIG. 1 is a block diagram of the media broadcasting system
compresses the content (e.g., in WindoWs Media Video,
MPEG-4, or H.264), and streams the content over an IP 45 in accordance With one embodiment of the invention. As
netWork (such as IP4 or IP6) to a client device for vieWing
and/ or listening to the content. The personal broadcaster,
Which may be implemented as an embedded solution, may
alloW the user to issue control commands, such as “channel
up” via an IR or serial command, back over the netWork to be
illustrated, a personal media broadcaster 100 is con?gured to
receive an input video signal from a Wide variety of A/V
source devices 120. For example, any component or device
50
from a connectedA/V source device 120, the personal broad
caster digitiZes, encodes, and streams the digitally com
executed by the original source device. Using various
embodiments of the present invention, a user can connect to
the personal broadcaster a cable or satellite set-top box, a
personal video recorder, a DVD player, a DVD jukebox, a
music server, a satellite radio receiver, a camcorder, a digital
video recorder (DVR) or any other A/V source component.
pressed media to the home Internet gateWay 11 0. The gateWay
55
a single device that encompasses one or more of these func
60
media player may be a softWare application that runs on a
client device, Which is con?gured to receive the media stream
created by the personal broadcaster. Different client softWare
applications can exist for different classes of client devices,
such as personal computers and cell phones. The client may
be able to connect to the personal broadcaster through a Web
broWser interface.
110 may comprise one or more separate devices, including a
router, a sWitch or hub, and/or an analog, DSL, cable or other
type of broadband modem, or the gateWay 110 may comprise
This Way, the user can vieW and control the live output of
these sources from any netWorked device. Various embodi
ments of the present invention may include different compo
nents, including a personal broadcaster and media player. The
having analog A/V outputs can be connected to the personal
broadcaster 100. Upon receiving the video and/ or audio feed
65
tions.
The gateWay 110 may be coupled to a local area netWork
(LAN) 140 that couples several computing devices in a user’ s
home. According to knoWn techniques, any number of local
clients 150 may be able to communicate With the home Inter
net gateWay 110. In this Way, created by the media broad
caster 100 may be routed to any of these local clients 150 by
Way of the local netWork 140, either through the gateWay 110
or directly. The local area netWork 140 can be Wired or Wire
less, as the present invention is not limited to any particular
US 7,877,776 B2
5
6
network technology or con?guration. The local clients 140
can be any number of device types, including but not limited
ing passWord before being automatically directed to their
personal broadcaster 100. This can be accomplished by tying
to desktop and notebook PCs, Web tablets, PDAs, embedded
clients built expressly for the purposes of decoding the
streams of the personal broadcaster, and other devices
dynamic IP address. The translation betWeen the device user
a unique device name to the DNS name assigned to the user’ s
name and the DNS name can take place Within the remote
client 170 itself, or it can be accomplished through a directory
capable of receiving and/or playing a media stream over a
netWork.
The media streams created by the personal broadcaster 100
maintained on the central server 180.
Connections from a local client 140 or a remote client 170
may also be received by remote clients 170 from a remote
netWork 160. The remote netWork 160 may comprise any
can be accomplished either by using a client application
designed speci?cally for the purpose of accessing the per
suitable netWorking technology, including but not limited to
Wide area mobile netWorks (e.g., GPRS, EDGE, lX-RTT,
lx-EvDO, and FOMA 2.5G and 3G cellular netWorks), WiFi
The option of using a Web broWser provides for Wide range of
client compatibility Without the need for con?guration, While
sonal broadcaster stream or via a traditional Web broWser.
the client application provides for an optimiZed experience.
and other public broadband access locations, WiMAX net
Works, other LANs (such as at Work, school, or a friend’s
home), and direct connections to other Internet service pro
viders. As With the local clients 150, the remote clients 170
may include any number of device types, but not limited to
desktop and notebook PCs, Web tablets, PDAs, embedded
clients built expressly for the purposes of decoding the
The client application or the Web interface may prompt the
user for a passWord before alloWing communication With the
broadcaster 100, as a security measure. As an additional mea
sure of security, the media stream can be encrypted.
In one embodiment of the invention, there is a limit of one
25
connected client (applies equally to remote clients and local
clients) per device. That is, only one client at a given point in
time can be connected to and streaming from the personal
broadcaster. Other variations of this embodiment can provide
for multiple simultaneous sessions. Still other variations can
alloW for multiple simultaneous sessions from local clients,
Web broWser. When a remote client 170 Wishes to connect to 30
but maintain a single session limit for remote clients.
FIG. 2 illustrates an embodiment of the personal media
broadcaster 100 having an interface for receiving a video
signal from a collection of possible A/V source devices 120.
The personal broadcaster 100 can thus support a number of
20
streams of the personal broadcaster, and other devices
capable of receiving and/or playing a media stream over a
netWork.
In one embodiment, the local clients 150 and/ or the remote
clients 170 execute a client softWare application that includes
a user interface for requesting content from the broadcaster
100 and for vieWing that content. In another embodiment, the
client functionality is provided by a Web site and is accessible
by the local clients 150 and/or the remote clients 170 via a
the stream of the personal broadcaster 100 using the client
application or via a Web broWser interface, it may specify the
input types and possibly may include a number of outputs
types, according to one embodiment of the present invention.
In the embodiment illustrated in FIG. 2, the personal broad
home IP address of the user to access and pull the media
stream from the personal broadcaster. This action sends a
request to the personal broadcaster, and the request travels
caster 100 can support a composite video input 210, an
35
across the public Internet, to the netWork of the user’s Internet
service provider (ISP), into the home via the telephony or
cable infrastructure (or Wirelessly in the case of the ?xed
Wireless or satellite broadband ISP), to the home Internet
gateWay 110, and ?nally to the personal broadcaster 100.
40
be provided for the same purpose. A Wide variety of video and
audio inputs are possible, in addition to those shoWn in FIG.
In one embodiment, a central server 180 is coupled to the
remote netWork 160 and provides one or more roles, includ
ing that of DNS server. Because most residential ISPs allocate
2. Inputs and outputs can be either analog (e.g., component
video) or digital (e.g., DVI, IEEE 1394, USB, SDI, Toslink
dynamic IP addresses via DHCP, as opposed to providing
static IP addresses, there is a need for a system that provides
a consistent method for accessing the user’s home netWork
from remote netWorks. In the embodiment illustrated in FIG.
S-video video input 200, a coaxial cable input 250, and left
and right audio inputs 220. The personal broadcaster 100 may
also have a coaxial cable output 260, Where the input cable
signal is split inside the personal broadcaster 100 to alloW a
pass through of that signal for local vieWing. Pass-through
outputs forA/V, S-video, and any of the other inputs may also
45
optical, or composite coax digital audio), and there may be
multiple connectors of a single type.
FIG. 2 also includes an IR output 270 and/or an RS-232
1, the central server 180 assigns the personal media broad
output 280. These outputs are intended to provide the ?nal leg
of backchannel control that originates from the client device.
caster 100 a DNS name (e.g., username.slingbox.com) and
correlates that DNS name to the user’ s IP address. To account 50 Depending on Whether the A/V source device 120 is con
trolled via IR or serial commands, the user connects an emit
for the dynamic nature of the IP address, in one embodiment,
a dynamic DNS client application resides on the personal
ter cable from the appropriate output on the personal broad
broadcaster 100. The dynamic DNS client application reports
caster 100 to a serial input or the IR receiver on theA/V source
to the central server 180 any change to the IP address leased
by the ISP. When a remote client 170 needs to communicate
device 120. This provides the communication means that
alloWs the client to control the A/V source device 120 (e.g., to
55
With the broadcaster 100, the client 170 ?rst obtains the
change the channels).
associated IP address from the central server 180. In this Way,
In FIG. 2, the personal broadcaster also includes an Ether
net port 290 that provides a communication interface to the
home Internet gateWay 110. In some embodiments of the
the DNS address called by a user on a remote client 170 is the
current IP address of the gateWay 110, even When that address
changes over time.
60
To make this process even easier for the user, so that the
user need not manage a constantly changing IP address or
enter a DNS name, much of the process for connecting to a
personal broadcaster 100 can be abstracted from the user. For
example, in one embodiment, the user need only to enter the
name of the personal broadcaster, or select an icon represent
ing the personal broadcaster, and then enter the correspond
invention, the personal broadcaster 100 also supports Wire
less netWorking (e.g., through a built-in 802.11 capability),
and the broadcaster 100 may even be built as an access point
(AP) or router for a Wireless netWork according to a Wireless
netWorking standard such as 802.11. The personal broad
65
caster 100 can also include a poWer connector 230, a hard
reset button 240, and a number of indicator lights (e.g., on a
front panel) that shoW the state of the personal broadcaster
US 7,877,776 B2
7
8
100. Many other inputs and outputs are also possible. For
example, the personal broadcaster 100 can have video and
analog outputs for a local display and sound.
processor 320 is capable of statically and/or dynamically
adjusting the compression bit rate, frame rate, resolution, and
FIG. 2 also shoWs the connections possible betWeen an
A/V source device 130 and one embodiment of the personal
broadcaster 100. As illustrated, an S-video cable and left and
able netWork bandWidth, or on any other data knoWn to the
processor 320. The compressed media stream is then con
verted into netWork packets for transmission over the local
right composite audio cables connect the personal broad
netWork 140 or a remote netWork 160 via the netWork inter
face 325. The netWork interface 325 may be a Wireless or a
?ltering depending on a user request, the input content, avail
caster 100 and an example A/V source device 120. Because
the A/V source device 120 in this example is controlled via
IR, controls called for by the remote client 170 or local client
150 are relayed from the personal broadcaster 100 to the A/V
Wired interface, or may have provisions for both types. As
mentioned above, the personal broadcaster 100 may also
source device 120 via an IR emitter 285 (e.g., an IR blaster
netWork interface 325. Some examples of these commands
assembly). One end of the IR emitter 285 is plugged into the
personal broadcaster 100, Which “blasts” appropriate IR
include selecting a particular channel, automatic scanning of
channels, sWitching betWeen RF input and base-band input,
codes through a Wire and out an IR LED of the IR emitter 285.
changing compression properties (compression type, bit rate,
Accordingly, the IR emitter 285 is placed directly in front of
frame rate, resolution, and other properties), remotely con
trolling commands for the IR blaster, and any other command
the IR receiver of the A/V source device 120.
The personal broadcaster 100 connects to the home Inter
net gateWay 110 from its Ethernet port 290 (using, e.g., a Cat5
cable), Which connection can be direct or via an Ethernet Wall
receive and process commands received from a client over the
that a user may desire for vieWing content from the A/V
source device 120.
20
jack located near the personal broadcaster 100 (Which in turn
connects to the home Internet gateWay 110). In other embodi
ments, the connection betWeen the personal broadcaster 100
and the home Internet gateWay 110 is Wireless, Where the
broadcaster 100 may include built-in Wireless or poWer line
A/V source device 120. As mentioned above, control of an
A/V source device 120 may be performed using an RS-232
serial controller or an IR emitter 285. The controller interface
25
networking capability.
FIG. 3 is a block diagram shoWing the internal components
of the personal media broadcaster 100, according to one
embodiment of the invention. As shoWn, the broadcaster 100
includes an input interface 305 for receiving any of a variety
of input types, including an RF signal from analog cable or
antenna, an S-video signal, a composite video signal, and a
335 thus receives the appropriate output signals from the
processor 320 and provides the corresponding interface for
controlling an operation of an A/V source device 120.
While FIG. 3 shoWs an embodiment of the broadcaster 100
30
that only takes analog inputs, other embodiments may accept
digital inputs as Well. For example, embodiments of the
present invention can be incorporated into service offerings
from cable MSOs or DBS providers. In these variations, the
personal broadcaster can have direct access to the digital bit
left/right audio signal. Because an RF signal includes a num
ber of video signals modulated therein, the input interface 3 05
is coupled to provide the RF input to a tuner 310. The tuner
310 ?lters the RF signal for a selected channel, demodulates
The broadcaster 100 may further comprise a controller
interface 335 for interfacing With an output for controlling an
stream being broadcast. This can be through incorporation of
35
the personal broadcaster into a set-top box or home Internet
gateWay by the service provider, or through some digital
the channel, and converts the signal into separate analog
interconnect such as IEEE 1394 or USB 2.0. With access to
video and audio for further processing by an audio/video
decoder 315. The input interface 305 is coupled to provide the
S-video signal, a composite video signal, and a left/right
audio signal directly to theA/V decoder 315, as those signals
need not be processed by a tuner 310.
the digital bit streams, digitiZing and encoding/compression
of the streams can be entirely unnecessary. For these inputs,
40
through a home Internet gateWay and out to the public Inter
net for connection by remote clients. In variations that include
In one embodiment, the A/V decoder 315 converts the
analog video input into YUV video and applies various types
of ?lters and color corrections to the signal. The A/V decoder
hoWever, the personal broadcaster can be asked to transrate or
transcode the media to a bit rate that is loW enough for effec
tive distribution across local area netWorks and upstream
45
digital inputs, analog inputs and a digitiZation and encoding
315 also extracts vertical blanking interval (VBI) data, such
as close caption, tele-text, and copy protection bits. The A/V
function can still be present.
decoder 315 also decodes the tuner audio and converts it into
stereo or mono digital audio, depending on the broadcast
there are many variations on the present invention that have
signal. The analog signal is further converted into a digital
signal in the A/V decoder 315. The digital video and audio
Despite the speci?c netWork topology illustrated in FIG. 1,
the personal broadcaster placed in different positions relative
50
embodiment has the personal broadcaster incorporated into a
home Internet gateWay. By integrating With the router func
tionality, the broadcaster can completely automate the pro
cess of port con?guration (e.g., port forWarding). Altema
from the A/V decoder 315 is then sent to a processor 320 for
further processing. The personal broadcaster 1 00 may include
memory 330, such as ?ash memory or SDRAM, used by the
processor 320 for performing its associated processing tasks.
55
The memory 33 0 may also be used as a buffer for the outgoing
media stream, as described herein for various embodiments.
In one embodiment, the processor 320 performs pre-pro
cessing on the digital audio and video before compression.
The pre-processing can be performed based on the input type,
compression properties, and target bit rate. After pre-process
tively, if the home Internet gateWay is actually tWo or more
devices (e.g., a cable modem and a stand along router/sWitch),
the personal broadcaster can be located betWeen the cable
modem and the router. Both of these variations of the present
invention provide a unique quality of service opportunity.
60
Because the personal broadcaster is upstream from all net
Worked clients and can “talk” With the netWork router, the
video streams from the personal broadcaster can be priori
tiZed ahead of other, less time-critical traf?c. The control can
be applied both to tra?ic moving Within the local area netWork
ing, the processor compresses the audio and video to a desired
bit rate using any suitable compression technique (such as
WM9, MPEG-4, H.263, and H.264). The compressed audio
and video are multiplexed into a single media stream together
along With other user data such as close caption, tele-text,
parental control, and Macrovision. In one embodiment, the
to the other components. For example, one variation of one
65
as Well as to tra?ic moving out from the local area netWork to
the Internet. For example, a video stream coming from the
personal broadcaster can be granted higher priority than a
US 7,877,776 B2
10
print job request over the local network, or an email download
in CPU utilization would detrimentally impact the perfor
from a remote POP3 server. In either case, the personal broad
mance of the PC for accomplishing other tasks.
Because the personal broadcaster 100 provides the user
caster preferably incorporates a network switch as part of its
architecture.
As shown in FIG. 1, the personal broadcaster 100 may also
with access to the same A/V source devices 120 available at
home, it makes sense to provide the user with an interface to
theA/V source device 120 similar to the one used in the living
room setting. Most often, this is a handheld remote control. In
one embodiment, therefore, a “virtual” remote control is pro
be able to receive a digital audio or video stream or other
digital media from an on-network storage device 130. The
on-network storage device 130 may be a personal computer,
vided by the client application that includes either a generic
a networked attached storage device, or a dedicated media
server. For example, a user could have a collection of audio
and video clips stored on a personal computer or media server
that resides on the same home network as the personal broad
image representing the A/V source device’s remote control or
an image or likeness of the actual A/V source device’s remote
control. Moreover, the client application may support a num
ber of virtual remote controls, one customiZed for each A/V
caster. The user could then access the media on a remote client
source device 120. Interfaces on the client application are thus
170 over a remote network 160 by logging into the personal
broadcaster 100.
selected by the user to resemble each particular A/V source
device 120 found in the user’s home. For example, if a TiVo
When wishing to stream media stored on an on-network
storage device 130 to a remote location, a challenge arises.
The bit rate of the media clips present on the on-network
personal video recorder is connected to the personal broad
caster, the user can decide to use the TiVo skin which modi?es
storage device can be higher than the bit rate supported by the
upstream link of the user’ s broadband service. For example, a
20
video clip on an on-network storage device 130 can have a bit
of the button triggers the action suggested by the image of the
rate of 800 kbps, whereas very few broadband connections
currently have an uplink speed equal to or greater than that. In
such cases, the bit rate of the source media signal is reduced
and its encoding format is possibility changed. The personal
broadcaster 100 may perform this transrating and transcoding
functionality. In this situation, the personal broadcaster 100
button (e.g., pressing on CH+ button turns the channel from 3
to 4).
25
likely to update the client application upon con?guration to
acts as a networked-attached transrater and transcoder. The
ensure that the latest remote control skins are available to the
30
the image. In this case, a method for “plugging-in” the third
proper transrating or transcoding settings (e. g., the bit rate to
35
transrate and/or transcode the material residing on one or
more on-network storage devices 130 has the bene?t of cre
allow users to develop content as well as third party device
manufacturers and service providers to make virtual remote
controls that closely resemble the physical remote control
40
ating a system where only one device (the personal broad
caster) streams media upstream through the home Internet
gateway 110 and out to the Internet 160. This is bene?cial
because the user does not have to make further con?gurations
to the NAT or ?rewall of the home Internet gateway, which
can include manually forwarding a port to allow direct access
to each on-network storage device.
In addition to requiring further con?guration, streaming
associated with anA/V source device 120. The virtual remote
on the client application can thus be made easier to use, since
the user is used to its layout.
Operation of Personal Media Broadcasting System
45
As described above with reference to FIGS. 1 through 3,
the personal media broadcaster 1 00 can receive an input video
signal from any of a number of A/V source devices 120. The
broadcaster 100 then prepares the received video signal as a
media stream for being transmitted over a network to a remote
50
content directly from the on-network storage device 130 can
create security concerns for the user, especially if the on
network storage device is a PC. Because allowing a PC to
stream directly up stream to the public Internet involves open
ing a port on the ?rewall/NAT that forwards to the PC, a user
party remote control skins is provided to the user. To allow
third parties to create skins for the media player, an API is
provided to allow access to some of the features and func
tionality within the media player client. Third party skins
plished within the framework for determining the throughput
currently supported between the personal broadcaster and the
local or remote client, outlined below.
The use of the personal broadcaster 100 as an agent to
users. In addition, third parties may be allowed to create and
share images with commands mapped to particular regions of
media, and outputs a media stream with a suf?ciently low bit
rate so the media can be effectively streamed upstream from
the user’ s broadband service. The method for determining the
which the source content is to be reduced) can be accom
In one embodiment, the client application contains a data
base of remote control skins from which to choose. In this
case, a central database maintained on the central server, is
broadcaster receives the media stream from the on-network
storage device 130, transrates and possibly transcodes the
the virtual remote control on the client application to
resemble the TiVo remote control. Remote control commands
are mapped to the graphical image in such a way that a press
or local client, where the media stream is viewed by the user.
Additions, alternatives, and re?nements to this general pro
cess are described below.
Control of Audio/Visual Source Devices
As stated above, embodiments of the personal media
55
broadcaster allow a user to control anA/V source device from
can be concerned that other personal or private information is
clients connected to the remote or local networks. The client
at risk for being exposed. By relying on the personal broad
may allow for control of the user’s speci?c model of A/V
source device. Upon initial con?guration of the personal
caster to be a gateway for streaming media to the public
Internet, the PC or on-network storage device on which the
media is stored need not have a port forwarded to it. In such an
60
embodiment, the personal broadcaster is the only device for
which NAT port con?guration is required.
In addition, using the personal broadcaster instead of the
on-network storage device to transcode and transrate content
prevents the CPU on the on-network storage device 130 from
broadcaster and a client, the user indicates which make and
model of A/V source device the user would like to control
(e.g., TiVo Personal Video Recorder Series 2). The con?gu
ration software on the client then identi?es the group of IR or
65
serial codes that correspond to the speci?c A/V source in
question. This can be performed by searching a database that
ships with the included software, an updated database resid
being unnecessarily taxed. This is especially important if the
ing on the central server, or a database that resides on the
on-network storage device 130 is a PC, because the increase
personal broadcaster.
US 7,877,776 B2
11
12
In one embodiment of the invention, the IR codes is then
caster to send a set number of bits to the client (this can
stored on the client device. When a user Wishes to invoke a
happen automatically at ?rst connection, or can be manually
ordered by the user to recalibrate initial optimization). Based
code, the user selects the given command on the control panel
or virtual remote control in the client application. The client
application then sends the corresponding IR or serial com
on the time it takes for the client to receive those bits, the
client has an idea of the actual data throughput betWeen the
personal broadcaster and the client. With this information in
hand, the client instructs the personal broadcaster to begin
streaming at a rate compatible With this throughput. This is
mand over the IP netWork. Once the IR or serial command
reaches the personal broadcaster, the personal broadcaster
processes the code and sends it out the IR or serial output,
triggering the requested action in the A/V source device. The
connection betWeen personal broadcaster and the A/V source
not likely to be the full rate of throughput, but some rate less
device can be an IR emitter, in the case that the A/V source
device Was to be controlled via IR, or an RS-232 port, in the
case that the A/V source device Was to be controlled by serial
variation in netWork bandWidth. In choosing the proper reso
(for example, 80% of throughput), to alloW for inevitable
lution setting, the application residing on the client notes its
current capabilities (i.e., resolution of its display) and pairs
the appropriate bit rate setting With the appropriate resolution
setting in its command to the personal broadcaster.
commands.
In another embodiment, the IR and serial codes are stored
on the personal broadcaster rather than the clients. In this
The client can learn of its capabilities in several Ways. One
can be related to the version of the application itself. For
example, the application for a Pocket PC can knoW that the
case, a client device that requests a given command sends a
notation representing that command (e.g., “CH UP” if the
user Wants to change the channel from 4 to 5) rather than the
IR or serial control code itself. When the notation represent
device best supports streams at or beloW a certain resolution
20
ing the command reaches the personal broadcaster, the per
sonal broadcaster performs a simple lookup, and outputs the
appropriate IR or serial code. Certain commands or selections
made on the client device can activate a series of commands,
also knoWn as a macro. For example, by clicking on the
25
can identify its display resolution and incorporate this infor
mation in the streaming request to the personal broadcaster.
The initial optimization process represents a starting point
that can very Well provide for the proper encoder settings.
HoWever, the variability of netWork bandWidth over time calls
for a system that is dynamic in nature and capable of real time
changes to the encoder settings. To address this variability,
30
back loop betWeen the client and personal broadcaster to
“CNN” button (Which can possibly be represented by a logo
of the network), the combination of commands that tune the
A/V source device to CNN Would be triggered. For example,
one embodiment of the present invention implements a feed
if CNN Was channel 202 on a user’ s DirecTV system, a press
of the “CNN” button can trigger the folloWing commands in
succession: “2,”“0,”“2,”and “Enter.”
maintain the proper encoder settings over time. This feedback
loop can be implemented in a number of Ways. In one embodi
ment the client gives noti?cation to the personal broadcaster
To make the set up of multiple client devices easier, pro?les
of the AV source devices used can be stored in the personal
broadcaster during the con?guration of the initial client
device. This enables easy con?guration for subsequent client
devices, as the personal broadcaster informs the subsequent
When it experiences frame drops. Alternatively, the client
35
client device WhichA/V source devices it is able to connect to,
and Which IR or serial codes it uses.
Adjustment of Encoder Settings Based on Throughput and
Device Capability
or all of these statistics, or any other data that re?ects on the
40
45
broadcaster can reduce the encoding bit rate from 350 kbps to
280 kbps if the siZe of the buffer becomes reduced or an
unacceptable number of frame drops are observed. The time
betWeen measurements and feedback received from the client
can vary, and the ?gure depends on a balance betWeen react
by a variety of client device types connected to the local area
netWork as Well a various remote netWorks, the available data
end and the local clients and remote clients on another can
vary considerably based on netWork topology. There is also
likely to be considerable throughput variation in a given con
nection, due to competing traf?c and general netWork con
gestion. In one embodiment of the invention, a method for
ing quickly to signi?cant changes and overcorrecting based
on temporary blips. Based on the feedback, the personal
broadcaster can adjust the settings upWard (e.g., increase
50
optimiZing the audio (e.g., bit rate and sampling rate) and
video (e.g., bit rate, resolution, and frame rate) compression
client device is implemented.
“quality,” and time betWeen key frames for video, bit rate and
55
sampling rate for audio, as Well as client side settings that can
impact performance, such as buffer siZe and smoothing.
Adjustment of Encoder Settings Based on Programming Type
While the data throughput betWeen the personal broad
at a time), the tWo components are able to act in concert to
optimiZe the experience for the user. In one embodiment, the
optimiZation process includes an initial optimiZation stage
and an ongoing optimiZation stage.
In the initial optimiZation process, the client and personal
broadcaster communicate to mutually establish the capabili
60
ties of the client device, as Well as the throughput of the
65
connection betWeen the personal broadcaster and the client
device. The client device ?rst requests the personal broad
encoding bit rate) as Well as doWnWard.
While one embodiment provides for a method of automatic
adjustment of encoder settings, the user may also have the
ability to set the encoder settings manually. These manually
adjustable settings include frame rate, bit rate, resolution,
based on available netWork bandWidth and capabilities of
Because various embodiments of the present invention
encompass both the personal broadcaster and the client
devices, and these elements can operate in a 1:1 relationship
(i.e., each broadcaster may accept only one client connection
communicates, from time to time, the siZe of the buffer, or the
total measured throughput. The client may communicate any
need to adjust the settings.
Given this feedback from the client, the personal broad
caster alters the encoder settings. For example, the personal
Because the broadcaster enables access of a media stream
throughput present betWeen the personal broadcaster on one
and frame rate. Another Way is to take inventory of system
resources before it sends the request. For example, the client
caster and the client may be one important determinant of the
proper encoding settings, the type of content being vieWed
may also be an important criterion. For example, fast motion
video from a sports program requires a higher frame rate than
a talk shoW, Which features much less motion. LikeWise,
video With little movement can require comparatively loWer
bit rates or resolution than fast motion video to achieve an
acceptable quality. This is because video With sloWer move
US 7,877,776 B2
13
14
ment tends to be encoded much more e?iciently. Accordingly,
the encoder settings may be selected based on the type of
bandWidth goes beloW minimum bandWidth required by the
broadcaster for extended period of time, a larger buffer helps
programming being encoded.
to reduce this risk.
As the data for the media stream are being generated by the
There are various methods for determining the type of
content being vieWed, and hence the proper range of encoder
broadcaster, the intermediate buffer acts as a FIFO queue.
When the available netWork bandWidth is more than the
settings. First, there can be settings that apply generally to all
content on a given channel. By identifying the programming
encoder’s bandWidth, the broadcaster is able to send data as
soon as it is generated. The intermediate buffer Will begin to
empty. When the available netWork bandWidth is less than the
channel or netWork to Which the personal broadcaster is tuned
(e.g., HBO or NBC), the client can request suitable encoder
settings. For example, there can be a rule that When the
encoder’s bandWidth, the broadcaster Will generate the data
personal broadcaster is streaming content from ESPN, the
faster than it can be transmitted. This Will begin to ?ll the
buffer. The buffer may therefore vary betWeen being com
frame rate is alWays set at 30 frames per second. Such rules
pletely full and completely empty. To classify the occupancy
can be stored on the client or on the central server, Which
informs the client of the proper settings for a requested chan
of the buffer, a number of Watermarks are de?ned to indicate
the amount of free space left in the buffer. While the number
nel. Moreover, the rules can be learned over time by an indi
of Watermarks can vary, in one embodiment four Watermarks
vidual’s oWn client, Which observes the settings chosen by the
are usediat the 90%, 75%, 50%, and 30% levels. As data are
added to and taken from the buffer, the amount of data ?lling
user for certain netWork programming.
In a further re?nement, the encoder settings may be cus
tomiZed based on the speci?c programming being Watched.
the buffer can change over time. When this level reaches one
20
Because a major netWork typically has content that is both
demanding, high-motion programming (such as a major
sporting event) and easy, loW-motion programming (such as a
neWscast), selecting encoder settings based solely on the
channel may not be e?icient for many channels. Accordingly,
by cross referencing the current channel With the current time
of day, the system can determine the program that is being
vieWed, and the encoder’s settings can be selected based on
the actual program being vieWed.
of the Watermarks, various actions are taken depending on
Which Watermark has been reached.
The amount of free space left in the intermediate buffer is
observed for a period of time (e.g., one minute). If the amount
of free space in the buffer remains above the 90% Watermark
during the last observation period, the encoder’s output bit
rate may be increased. Although any increase can be imple
mented based on the application, in one embodiment the
increment is about 10% of the bit rate then being used.
Because netWork bandWidth varies over time, sudden
30
In another embodiment, the system constantly monitors
the A/V content being encoded. Metrics based on pre-selected
criteria (e.g., amount of motion in video) are generated, and
drops in available bandWidth may take place frequently. In
such cases, the TCP stack transmit rate Will go doWn and the
occupancy of the intermediate buffer Will increase. If this
occurs for a long enough period, the free space in the buffer
this information is used to assign or adjust the encoder set
the media player may receive from the user an indication of
Will decrease so that the 90% Watermark may be breached. In
response, the broadcaster reduces the encoder’s bit rate by a
small percentage, for example about 15% of the bit rate then
What kind of content is being Watched (e.g., action, music,
neWs, etc.), Which is mapped to prede?ned pro?les in the
being used. If this netWork problem is a temporary one, the
TCP stack Will again be able to send the backlog of data in the
tings to the personal broadcaster dynamically. Alternatively,
35
framework optimiZed for that kind of content.
Buffering and Control of Buffer Resources
40
In accordance With one embodiment of the invention, the
buffer so that the amount of free space in the buffer Will rises
again above the 90% mark. The encoder’ s bit rate can then be
increased.
On the other hand, if the netWork problem persists, the
personal media broadcaster implements a buffering scheme
amount of free space Will continue to decline. Over a period
to manage its buffer resources Without requiring feedback
from a client device. As mentioned above, the broadcaster and
the client may communicate using TCP as a transport proto
col, Where the broadcaster acts as a server. Bene?cially, TCP
is a reliable protocol and ensures that sent data alWays reaches
its destination in the correct order. Parameters and/or behav
of time, the buffer Will ?ll and the other Watermarks Will be
breached. As each Watermark is breached, the encoder’s bit
rate is further reduced. In one embodiment, these subsequent
reductions may be larger (e.g., 33%, 50%, 50% for each
Watermark, respectively).
As described, the system intelligently exploits TCP stack
ior of the TCP stack on a server can be monitored to estimate 50
behavior to estimate netWork status and reacts to provide
netWork congestion and speed according to one or more
optimal user experience in presence of bandWidth variations.
This may offer improved performance as compared to using
knoWn techniques.
In accordance With an embodiment of the invention, a large
buffer is added betWeen the encoder (Which generates the
data) and the TCP stack on the netWork interface (Which
transmits the data). This additional buffer layer added above
client-server interaction, Which can be complicated, react
more sloWly, and may not make correct decisions if the
55
Fast-Start Streaming
the TCP stack helps to avoid loss of data due to netWork
congestion and the variability of data rates. In one embodi
ment, With reference to FIG. 3, the encoder functionality is
Conventionally, When a streaming media player receives a
command to play a media stream, the media player ?lls its
audio/video buffers before starting the playback. A buffer of
?ve seconds or longer is typically maintained to ensure
smooth playback of the media, since the time taken to transfer
performed by the processor 320, the TCP stack functionality
performed by the netWork interface 325, and the buffer layer
implemented in the general memory 330 or in a memory
module dedicated for the large buffer. The siZe of this buffer
the media stream over a netWork typically varies While the
media stream is meant to be played at a constant rate. Once the
can be selected in consideration of at least tWo parameters: the
minimum data that can be generated by the encoder and the
maximum netWork doWn time that has to be supported.
Although the system cannot prevent data loss When available
encoder’s output bit rate is expected to vary.
65
required amount of data is accumulated to ?ll the buffer
suf?ciently, the media player starts playing the requested
content at normal play speed (i.e., l.0><). Disadvantageously,
US 7,877,776 B2
15
16
the user must Wait the amount of time required to ?ll the
the server and the client. This delay is needed to ?ll the A/V
buffer before vieWing the requested content. This time may be
very small in applications that send the stream faster than real
time (e.g., a media stream from a storage device), but it is
noticeable When the received media stream is at normal play
back speed (i.e., 1.0x). In such a case, ?lling a ?ve-second
buffers in the media player client. The tWo requirements of
loW delay for user interactions and smooth audio-video dis
play are con?icting in nature. To deal With both of these
requirements, in one embodiment of the invention, tWo
modes of operations are introduced: normal mode and control
mode.
In the normal mode, the system performs conventional
buffer Would take ?ve seconds (assuming no netWork com
munication issues). While this delay may be tolerated in some
streaming media applications, it becomes unbearable Where
audio-video streaming, Wherein a buffer of ?ve or more sec
the media streams are changing, such as in a personal media
onds is maintained by the media player to ensure smooth
playback. In this mode, the media broadcaster may also start
buffering data if the netWork bit rate drops beloW encoder bit
rate (i.e., the encoder at the broadcaster runs ahead of the
media stream transmission). The total delay betWeen video
input to the media broadcaster and its vieWing on the media
player is thus the sum of three parameters: the time taken to
buffer on the media broadcaster, the netWork transmission
broadcaster in Which the vieWer is changing channels. In such
a case, the user Would have to Wait for the buffer to re?ll each
time the channel (and thus the media stream) is changed.
To avoid this disadvantage, in one embodiment of the
invention, the media player performs a fast start When a neW
media stream is selected. With the fast start, the media player
client application plays the video immediately as it is received
from the media broadcaster While still ?lling its buffer. In this
Way, the user does not have to Wait for the media player’ s A/V
buffers to ?ll up, e.g., upon channels changes, and the media
player can still build its buffers to provide smooth playback.
The media player is able to ?ll its buffers even though it plays
the media stream immediately because the media player plays
the A/V stream slightly sloWer than normal play speed. By
playing the stream slightly sloWer than normal playback
20
be directly controlled; hoWever, both the media broadcaster
and the media player can minimiZe delays caused by their
25
speed, the part of the received media stream that runs ahead of
acting With the media broadcaster by Way of the media player.
accumulation of the A/V bufferialbeit more sloWly. Because
In one embodiment, user interaction is de?ned as When the
user requests an IR command or tuner command to control the
the user need not Wait for content to be buffered and can
30
much more similar to that With a normal television.
In one example, the media player begins in fast-start mode
35
media player sloWly accumulates media stream data in its
A/V buffer. For example, if the playback speed is 85% nor
mode, as described beloW.
On entering control mode, the media player and the media
40
accumulating data.
When the play speed of stream is changed slightly, the
45
ever, the change in audio becomes immediately perceptible.
To take care of this, the media player may use time-stretching
on the audio stream While maintaining the “pitch” in the audio
stream. One softWare tool that can be used to time-stretch the
audio stream is SoundTouch, an open-source audio process
50
2-3 seconds Worth of content. SP then ?ushes all the data
55
60
streaming speed and the initial fast-start playback speed.
Control Mode For LoW Latency
One of the most used features in TV vieWing is channel
control, and users expect such control operations to take at
most one second to execute. But streaming video over IP
netWorks performs best When some delay is added betWeen
the media stream from the netWork buffer, parses the stream,
and ?lls its audio/video buffers. This buffer is maintained to
ensure smooth playback. When going into control mode, the
media player ?ushes the data that is present in the A/V buffers
and the netWork buffer. In one embodiment, When going into
control mode, the media player makes the source ?lter ?ush
all the data buffered in the A/V buffers and also in all the ?lters
present in the netWork buffer. Thereafter, the media player
playback speed. The timestamps of the audio and video
samples may be changed according to the existing playback
speed to reduce jerks in the video stream. As a result, the
change in the audio stream is also not perceptible, and the user
is less likely to perceive the difference betWeen the normal
mode in a number of Ways.
doWnstream (decoders and renderers), Which may be holding
ing library. In one embodiment, the playback speed is
increased gradually from the loW threshold (e. g., 0.85><) to the
real time playback speed (e.g., 1.0x). The rate at Which the
playback speed is increased may be a function of the buffer
level, so the user does not perceive any drastic change in the
broadcaster change their behavior relative to that in normal
As mentioned, in the normal mode the media player per
forms normal A/V streaming Wherein the media player reads
the media stream at a normal rate, and the A/V buffer stops
change in the video stream is generally not noticeable. HoW
normal mode from control mode after a predetermined
amount of time has passed since the last action that could
cause the system to start control mode. In another embodi
ment, the system returns to normal mode immediately after it
performs the operations associated With going into control
mal, the data accumulates in the A/V buffer at a rate of 15%
(or 1.5 second for every 10 seconds of received streaming
data). Once the media player’s buffer is full or otherWise
reaches an acceptable level, the media player begins to play
operation of the A/V source device through the media player
user interface. In one embodiment, the system returns to
playing back a received media stream immediately but at
slightly sloWer speed than normal play speed, e.g., 0.85><
speed. Since the broadcaster is streaming at normal speed, the
buffers.
The system enters control mode When the user starts inter
the stream played back is added to the buffer, resulting in
immediately start Watching the content, the experience is
time taken for the stream to move from the media broadcaster
to the media player, and the time taken to buffer on the media
player. The netWork transmission time taken for the stream to
move from the media broadcaster to the media player cannot
65
sends a noti?cation to the media broadcaster to go into control
mode and Waits for the next I-frame received from the media
broadcaster.
After the ?ush operation on the source ?lter and the net
Work buffer, discontinuities occur in the media stream. The
?lter has intelligence built in so that if there is any packet
discontinuity in the media stream, the ?lter Waits for the next
I-frame. Any incoming data that is not an I-frame is discarded
by the source ?lter until a valid I-frame is detected. When the
media broadcaster goes into control mode, it sends an I-frame
immediately. As soon as this I-frame is detected by the source
?lter, it is sent doWnstream for rendering. While the media
player remains in control mode, no buffering occurs in the
source ?lter; the samples are sent doWnstream for rendering
US 7,877,776 B2
18
17
Trick Play of Received IP Stream
as soon as they are read from the network buffer. In this
manner, the media player reduces the latency on the client
side.
In one embodiment, upon a change into control mode, the
media broadcaster stops buffering data and ?ushes data cur
rently contained in its buffers. The media broadcaster then
In one embodiment, a user vieWing a live stream being
encoded in real time by the personal broadcaster and dis
played on the media player can replay the last several seconds
or minutes of content just vieWed, or it can pause the live
stream for resumption at some point in the near future. The
personal broadcaster receives and processes the analog input
immediately generates an I-frame (also knoWn as key frame)
signal; converts it into digital format; encodes the video in a
to send to the media player. The I-frame alloWs the media
player to reconstruct an image, Whereas other types of frames
that encode the frame based on previous frames Would not
alloW the media player to reconstruct the frame due to the
discontinuity in the media stream.
compression algorithm such as MPEG-4, H.264, WindoWs
MediaV1deo Series 9, or another appropriate format; and then
streams the encoded via stream over TCP/IP (or an alternative
protocol, such as UDP, RTP, RTSP) to the media player.
To enable this functionality, the media player caches the
Given the dependency of modern audio-video encoding
last ?ve minutes (or some other ?xed period of time as chosen
standards, hoWever, it may not be desirable for the media
broadcaster to stop buffering data completely. This is because
by user or limited by product manufacturer) of the audio and
video content to a local storage medium While receiving a
video stream. The media player may store the data on a local
storage medium, such as a hard drive in the case of a PC, or
Without buffering there maybe too much data loss, leading to
an extremely poor user experience. A compromise can there
fore be made to balance loW delay and a reasonable user
experience. Based on a set of empirical values for achieving a
20
good balance, an amount of data is alloWed to be buffered by
the media broadcaster during control mode as a function of bit
rate of content. For example, in control mode, buffered data
can be reduced by ?ushing the data if it crosses a limit of about
one second; hoWever, other empirical values may be used for
priate labeled button in the media player user interface. When
the user instructs the media player to reWind, the media player
accesses the content cached in its storage, alloWing the user to
scan through it, and play as desired. Once the user has
various applications.
“reWoun ” content, the user can then “fast forWard” the vieW
In one embodiment, the media broadcaster keeps track of
effective transmission bit rate, for example by periodically
calculating the average transmission bit rate for the last feW
seconds. Based on the bit rate it calculates permissible buffer
usage. If usage goes beyond the current limit then all data is
30
35
and then sends the data doWnstream for rendering.
To operate effectively in a loW latency, loW buffer environ
ment, a command can be immediately sent to the personal
broadcaster to reduce the bit rate When in control mode.
Without any additional adjustments, this Would result in a
to receive the incoming stream, Which it caches to its local
storage medium. The media player continues to cache the
stream as it is received, until the point When the maximum
number of minutes (or siZe of data) alloWed to be cached
When in pause mode is reached. If the maximum number of
minutes (or siZe of data) is reached, the media player resumes
playback. Otherwise, the media player resumes playback
force many I-frames, Which Will adversely affect the encoder.
Therefore, all buffer ?ushes and forcing of I-frames are pref
erably timed and spaced out by at least one second.
When returning to normal mode from control mode, the
source ?lter of the media player pauses the playback, buffers
(e. g., up to ?ve or more seconds) content in the A/V buffers,
ing of the stream at faster than real-time speed until the
current-most point in the video stream is being displayed.
When the user instructs the media player to pause, the media
player pauses the video stream being displayed but continues
removed and an I-frame is forced. This also means encoder
output rate is more than netWork transmission rate; therefore,
encoder output is reduced to half of observed transmission
rate. Due to the resulting discontinuity in the media stream, an
I-frame is also forced.
Too many user control commands in quick succession Will
some removable media, including but not limited to compact
?ash, smart media, a memory stick, or a micro drive.
When the user Wishes to pause or replay the content, the
user instructs the media player to do so by selecting an appro
When the user instructs the media player to do so.
Capturing, Editing, and Sending Video Clips From a Stream
ing Source
45
50
Embodiments of the invention also alloW people to send
video clips to friends and acquaintances. In accordance With
one embodiment, the media player device continually cap
tures and caches the last ?ve minutes (or some other desig
nated time period) of video and audio received. The media
player automatically discards the earliest recorded content
When the cached stream hits ?ve minutes (or some other
reduction in image quality; hoWever, because the menu
screens being navigated typically feature very little motion,
designated time period), so that the ?ve minutes being cached
the frame rate of the video can be signi?cantly reduced. With
less frames to process, the personal broadcaster can output a
When a user sees something of interest that the user Would
like to capture and save or share, the user clicks on or selects
stream With image quality good enough to effectively read the
is alWays the ?ve most recent minutes of media streamed.
55
a simple video editing interface, Which enables the user to
select the beginning and end of the clip that the user Would
like to capture. Once the bounds of the clip have been
The loW latency achieved by the dual mode operation of
this embodiment is a very desirable characteristic for systems
in Which the users interacts With the media broadcaster
through the media player. The addition of a control mode
achieves this loW latency, While the return to normal mode
60
selected, the user selects a command to save the clip or send
the clip to another person.
In response to a command to send the captured video clip,
the media player calls the email client resident on the user’s
once user interaction has stopped achieves the smooth
streaming desired for normal vieWing. Moreover, in combi
nation With the fast-start streaming described above, the con
trol mode function provides a loW latency control interaction
a designated button or command using the media player’s
graphical interface. Once selected, the media player brings up
on-screen text.
65
client device, creates a neW message, and attaches the video
clip to the message. The user can then select the intended
experience for the user With a seamless transition from con
recipients of the message and send the message. The address
trol mode to normal vieWing.
book databases can be those available from email clients such
US 7,877,776 B2
19
20
as Microsoft Outlook, Lotus Notes, and others, as Well as any
access the content from a remote client, the personal broad
caster acts as an arbiter betWeen the remote client, Which
Wants to vieW the content, and the on-netWork storage device.
The personal broadcaster receives the stream from the on
netWork storage device and transcodes or transrates the con
Internet based messaging services, such as MSN Messenger
and AOL Instant Messenger.
Personal Video Recorder (PVR) Functionality For Mobile
Devices
tent if necessary before repackaging and streaming it to the
As users get more accustomed to video experiences on
remote client.
mobile devices, and mobile storage solutions (e.g., removable
?ash media and small hard disk drives) become more robust
Pairing Programming Being VieWed With Context-Speci?c
and inexpensive, the desire to add personal video recorder
functionality to these devices Will emerge. One embodiment
of the present invention adds the functionality of a personal
Content/Advertisements
One embodiment of the invention provides the ability for
the client to present the user With Web pages, scrolling text
With neWs, or other information that varies based on the
video recorder to a mobile phone, communicator, PDA or
other device connecting to a Wide Area Network or other
content the user is currently vieWing. For example, if the user
is currently Watching a San Francisco Giants baseball game,
remote netWork outside a user’s local area netWork.
One variation that accomplishes this can feature a sched
uler as part of the remote client application Which coordinates
With an electronic programming guide. When the user
launches the remote client application, he can search for and
select the shoWs he Would like to have recorded to his remote
client. When the time for the scheduled recording occurs, the
the user can be presented With a “neWs ticker” that details
other baseball scores around the league, or perhaps a Web
page With statistics and facts about the San Francisco Giants.
The content can be embedded in the interface of the client
20
remote client application initiates a recording. The applica
tion can call the connection manager on the remote client,
Which in turn opens an Internet connection. The remote client
application then connects to the personal broadcaster at the
user’s home.
Next, the remote client application can issue the right set of
commands related to channel selection and encoder settings
on the server, and begins to capture the incoming stream onto
the local storage medium (e.g., hard disk drive or ?ash
memory). This can happen in the background, so a user can be
on phone call or listening to music (depending on other capa
25
application itself or canbe presented through the launching of
another application, such as a Web broWser. Similarly, the
user can be presented With content and context-speci?c
advertisements. An example in the context of the same base
ball game Would be an advertisement from the San Francisco
Giants ticket o?ice, Which hopes to attract vieWers to pur
chase tickets to a future game.
The broadcasting system can determine the content cur
rently being vieWed by the user in a number of Ways. In one
embodiment, the personal broadcaster or client sends infor
30
mation, including the current time, the channel lineup being
used (e.g., location and cable TV provider), and the current
channel being tuned to, to the central server. The central
bilities of the device) While the remote client is recording.
server then takes the information received and examines the
If the recording is interrupted because a netWork connec
tion is lost, the live streaming content can be temporarily
stored on the local (built-in or removable) storage that exists
electronic programming guide applicable to the user’s ser
35
vice. From information it is determined Which shoW the user
in one variation of the personal broadcaster. Once a connec
is currently vieWing. Alternatively, the client can perform the
programming guide lookup itself and transmit this informa
tion betWeen the personal broadcaster and the remote client is
tion to the central server.
After determining What shoW the user is currently vieWing,
reestablished, the content resumes the stream from the per
sonal broadcaster to the remote client, taking into consider
40
ation the point at Which the previously interrupted transfer left
off. In another variation, an on-netWork storage device, such
overlaid information and/or advertisements can be transmit
ted from the central server or other server on the public
Internet to the client media player. Alternatively, applications,
as a PC, can be used to store content temporarily after a
such as a Web broWser, can be launched, simply sending the
connection betWeen the personal broadcaster and the remote
client has been interrupted. In another embodiment, an entire
user to a speci?ed URL. The client can also contain the
information necessary to queue any relevant information or
45
shoW or media program is encoded and stored on the personal
broadcaster (or in a storage device), and then transferred to
the mobile device in the background. Bene?cially, the trans
fer can be performed at a more ef?cient time for data trans
fer4e.g., the ?le doWnloaded to a cellular phone overnight
50
during non peak hours so that the content is available to the
In the case Where the original source of the media being
tisements from a central server or other remote server.
using storage of the on-netWork storage device) can provide
streamed is an on-netWork storage device (using the personal
55
for context-speci?c ads and information even When playing a
previously recorded shoW. When recorded, the content is
“tagged” With information indicating the programming being
captured. On play back, this information can be used to
prompt content speci?c information and ads.
connectivity, the doWnload pick ups and continues later at the
proper spot in the content.
Another embodiment enabling a PVR on a remote device,
much of the need for transmission of content and or adver
Embodiments that include personal video recorder func
tionality (using local storage on the personal broadcaster or
user for of?ine vieWing the folloWing day.
broadcaster as a transcoder/transrater), transfers betWeen the
personal broadcaster and the remote client can be accom
plished over time. That is, if a remote client loses netWork
advertisement for display to the user. For example, the client
can have stored in its memory certain advertising banners or
URLs of relevant Web sites that are queued depending on
What type of content the user vieWs. This method alleviates
60
Content-Speci?c Edge Preserving Pre-Filter
LoW bit rate coding involving block based video coders
a variation of the embodiment described above, alloWs the
personal broadcaster to Work With an on-netWork storage
produce strong blocking artifacts. To reduce the severity of
device (such as a PC or a NAS or SAN device) to provide for
the user a PVR that can be accessed remotely. The personal
these artifacts, in one embodiment of the invention, pre-?l
tering is employed to simplify the image content before com
the content, and then streams the resulting media stream to the
pression. The pre-?lters often comprise loW pass ?lters,
Which not only smooth blocking artifacts but also undesirably
on-netWork storage device for storage. When a user Wants to
reduce image details. Moreover, loW pass ?lters are not adap
broadcaster takes in the analog content, digitiZes and encodes
65
US 7,877,776 B2
21
22
tive to video content, Which makes them ill-suited for percep
for all motion types. Unlike conventional pre-?lters this
tual video encoding. To avoid the limitations of previous
reduces the frame drops Without compromising quality.
pre-?lters, one embodiment of media player performs a con
Frame Rate Regulation and Quality Control For Encoder
tent-speci?c edge estimation algorithm on the media stream
Most video softWare encoders at loW bit rates dynamically
drop video frames to meet some speci?c target bit rate, often
during high motion. This dropping of frames can lead to jerky
received from the media broadcaster. In accordance With this
embodiment, an edge estimate gives the location of the details
in the image that should be preserved While a motion estimate
gives a classi?cation of Whether the content is in high,
video and ?uctuating quality levels. As potentially large sec
tions of frames are dropped, the motion estimation process
becomes ineffective. To avoid this problem, one embodiment
of the media broadcaster regulates frame rate using a multi
medium, or loW motion.
In one embodiment, the media player applies a pre-?lter on
the received stream. The pre-?lter comprises a loW pass ?lter
having pass band characteristics that are changed based on an
level approach designed to enhance the vieWing experience of
the user by sustaining both frame rate and quality at each
level.
In one embodiment, the encoder in the media broadcaster
edge estimate, a motion level estimate, and an encoding bit
rate. Based on the motion level estimate and the encoding bit
rate, a ?lter characteristic is de?ned for a frame. The ?lter
characteristic is then ?ne tuned at the pixel level based on the
estimates a sustainable frame rate based on a Sustainability
measure, SM. Four levels of frame rate regulation are de?ned.
Based on SM, the frame rate is selected and an appropriate
edge estimate. Using this approach, a higher degree of
smoothing is used for high motion contents and loWer degree
for loW motion content, While leaving the details intact.
In one embodiment, tWo sets of adaptive loW pass ?lters are
20
de?ned. For loW bit rate encoding, the folloWing Gaussian
quality. This results in better motion estimation and accept
able spatial video quality levels.
?lter of siZe 5><5 is used:
Frame rate regulation is performed in one embodiment
25
For high bit rate encoding, a loW pass ?lter having an average
mask of siZe 3x3 is used, as shoWn in the table beloW.
quality level is de?ned. Each stage of the algorithm keeps the
inter-frame distance constant, preserving temporal video
30
35
according to the folloWing algorithm. The target frame rate
de?ned by the encoder is F0. A sustainability measure, SM,
determines Whether the frame rate can be sustained over the
time interval T. SM can be de?ned as the ratio of the bit pool
available to the estimated bits required for the next T seconds.
The bit estimate is a product of frames per T seconds With the
recursive average of the bits per frame. The SM may be
calculated based on the “motion” estimate and the target bit
rate. The “motion” estimate is based on the recursive average
of the encoded bits per frame. In one embodiment, the frame
rates at subsequent levels are: F l:FO/2; F2IFO/3; and F3:FO/5.
If F,- denotes the assigned frame rate for level i, based on
SM, a decision is made every T Whether the encoder should
continue in the same level or move to a level higher or a level
loWer. If SM is less than a (indicating that that the current
frame rate, Fi, cannot be sustained for the time interval T), the
The parameters 0 and 11) are varied based on the edge estimate,
motion level estimate, and the encoding bit rate. Increasing o
or decreasing 11) increases the smoothness of the video, While
the opposite decreases the smoothness of the video. The edge
40
encoder moves to level i+l. If the encoder is already at the
loWest frame rate, F3, it stays there. This operation is again
performed after every T. The level thus goes doWn When bit
rate is not su?icient enough to cater high motion content bit
estimate, 6, is operated at pixel level and is obtained by taking
the gradient at all eight directions. This is given by:
rate need.
45
While in level i, as long as i is not 0, SM for level i-l is
checked after time interval of T. If the SM is greater than
[3(>l), it indicates that the current content can be encoded at
level i— 1, so the encoder moves to level i-l . This method thus
tries to maintain the inter-frame distances constant at each
If e is greater than a threshold T, no pre-?ltering is per
50
formed on the current pixel, thereby preserving the image
details. The motion level estimate gives a classi?cation of
Whether the motion is high, medium, or loW motion. The
estimate is based on the bits required for the current frame and
the recursive average of the bits per frame. If the current frame
bits are greater than 0t(alWays >1) times the recursive average,
the current video content is declared as high motion, and if the
current frame bits are less than [3(<l) times the recursive
average, the video content is classi?ed as loW motion. Other
Wise, it is classi?ed as medium motion. Based on the type of
Dynamic Parameter Control For Video Encoder
In a real time video streaming environment, netWork char
acteristics change dynamically. To improve the use of the
netWork resources, a video encoder should be able to adapt to
55
Media Video (WMV9) encoder is provided Wherein the
60
parameters can be changed dynamically during streaming.
The folloWing parameters can be changed dynamically
While streaming is in progress: bit rate, frame rate, video
smoothness, and I-frame interval. The bit rate can adapt to the
rate that can be supported by the netWork at a given time, and
applied as compared to loW motion content.
This pre-?ltering method not only retains the image details
the motion type: high, medium, or loW. This pre-?ltering
method provides non-blocky, constant quality encoded video
these netWork changes; hoWever, most of the standard video
encoders do not support such adaptation. In accordance With
one embodiment of the invention, a scheme for the WindoWs
motion, the ?lter strength is varied. For high motion content,
for example, a relatively heavy smoothing ?lter can be
but also is content speci?c. That is, the ?lters are adapted to
level thereby improving the overall video quality.
65
rate control buffer delay, bits per frame, and quantization step
siZe vary according to changes in bit rate. Frame rate depends
on the dynamically changed bit rate. At loWer bit rates, high
frame rates cannot be sustained, Which creates the need in
US 7,877,776 B2
23
24
some circumstances for a dynamic frame rate. The video
caster can thus instruct the NAT to forWard a speci?ed port to
smoothness parameter indicates the encoded video quality.
LoWer bit rates supporting high values of video smoothness
the internal IP address dedicated to the personal broadcaster.
If the home Internet gateWay does not support UPnP, the
system may then attempt to determine the type of NAT in the
home Internet gateWay, and speci?cally Whether the NAT is a
full cone NAT. This detection may be performed by using the
can cause jerky video, While loW values of video smoothness
at higher bit rates leads to under utilization of the available
resources. Depending on the video smoothness parameter, the
quantization upper bound and loWer bound step values are
central server as a STUN server, Which runs various tests to
changed, Which affects the quality of the video. Because the
bits required to encode I-frames are greater than bits required
determine What the type of the NAT behind Which the per
sonal broadcaster sits. There are four basic types of NATs: full
cone, restricted cone, port restricted cone, and symmetric. A
for P frames, I-frame intervals are larger for loWer bit rates.
I-frame intervals can be reduced at higher bit rates, Where
full cone NAT is a NAT that alloWs a client behind it to receive
messages from one external machine that are addressed to an
more bits can be afforded.
IP address and port that the internal client used in sending a
message to another external machine. If the personal broad
caster sits behind a full cone NAT, the folloWing is possible:
The personal broadcaster from time to time sends a message
Accordingly, the encoder in the media broadcaster can
dynamically adjust these parameters to adapt to the changing
netWork characteristics and optimiZe use of system resources.
This alloWs for improved performance than the standard
WMV9 encoder, Which does not dynamically change the
above parameters in a real time streaming environment.
Although described in the context of video encoding With
WindoWs Media Video (WMV9), this method can be applied
to the central server, and the central server makes a note of the
IP address and port from Which it Was received. When a
20
to other video compression formats, including MPEG-4,
H.263, H.264, and any other compression formats that use the
same or similar parameters mentioned herein.
Point and Click Interaction With Traditional CE Menus
Because the personal broadcaster digitiZes, encodes and
25
the central sever as a Way to “introduce” the remote client and
30
another simultaneously. Because both devices believe they
“Left,” and “Select” as its key components. HoWever, alter
possible. For example, instead of pressing the “Down” com
35
advantage of the rich human interface tools (e.g., keyboard
the present invention can Walk the user through the steps for
manual con?guration of the NAT on the home Internet gate
Way. To make this easy and integrated into the set up process
40
Hot Spot Finder
device and accessible When the client is not connected to a
netWork. Such a directory alloWs the user more easily to ?nd
a location Where the user can gain access to the personal
45
50
Testing and Con?guring a User’s NAT
For users that have a home Internet gateWay that includes a
55
be required to alloW a user to access the personal broadcaster
from a remote netWork. This is because incoming requests are
search and sort programming content by a number of vari
60
tion include a multi-step process for determining and imple
The system may ?rst attempt to determine Whether the
Play). If so, the personal broadcaster may be able to control
the home Internet gateWay using UPnP. The personal broad
ables, and a single “click” on a channel can automatically
tune the A/V source device and media player to the desired
channel.
Community or “Buddy” List
One embodiment of the media player incorporates a
menting the possible solutions.
home Internet gateWay supports UPnP (Universal Plug ’N
personal broadcaster. The user is asked for a Zip code and the
service and package to Which the user subscribes. Based on
this information, the client application doWnloads an EPG
that covers the next several days. From time to time, the EPG
is updated via doWnload from the central server or another
server from a third party provider. The EPG can be stored
locally on the client.
Bene?cially, the EPG can be made interactive. Among the
many features enabled by an on-device EPG, a user could
rejected by many NATs unless the NAT has been explicitly
instructed hoW to forWard the incoming packets. There are
many methods for solving this problem, some more desirable
and automated than others. Some embodiments of the inven
On-Device EPG
One variation of the present invention features an elec
tronic programming guide (EPG) that resides locally on a
client. The EPG is con?gured at initial con?guration of the
media broadcaster.
Network Address Translation (NAT), some con?guration can
for the personal broadcaster, the con?guration screens for the
home Internet gateWay can be embedded in the WindoW that
houses the set up application. This provides the user With a
greater sense of seamlessness.
and mouse) that are available in many computing devices but
not generally used With televisions.
In one embodiment of the invention, the media player
includes a directory of hotspots (Wireless broadband net
Works available in public locations) stored on the client
are “initiating” the conversation, the return packets are per
mitted to How through the NAT to the destination device.
If none of the above or other methods are viable solutions,
mand four times to highlight an item four spaces doWn from
What is currently highlighted, an embodiment of the media
player alloWs a user simply to point and click directly on the
desired menu item. This speeds up the interaction and takes
the personal broadcaster. The method Works for all NAT types
except for symmetric NATs (so the STUN test Would be
useful for determining if this is a viable option) and uses the
central server to cause both devices to send messages to one
of navigation typically involves “Up,” “Down,” “Right,”
native methods for interacting With menus and other lists are
client can use the same IP address and port to locate and
connect to the personal broadcaster. If the NAT is not a full
cone NAT, another method is used.
A third possible method to use comprises a UDP “hole
punching” technique. This technique, Which is knoWn to
those of ordinary skill in the relevant art, Would involve using
streams the analog output of the A/V source devices being
used, the client presents the user With the full interface of the
A/V source device. The default paradigm for navigating
menus of the A/V source device that are rendered by the client
is exactly the same as it Would be for a user vieWing the A/V
source device in a more traditional fashion. That is, the mode
remote client Wishes to connect to the personal broadcaster, it
asks the central server for the address and port recently used
by the personal broadcaster to send its message. The remote
65
“buddy list.” Using the buddy list, a user can connect to
personal broadcasters that reside at different locations. For
example, if Charlie declares Amy as a “buddy,” Charlie’s
personal broadcaster appears on Amy’s buddy list. By choos