Download Method of controlling a device to perform an activity

US007668990B2
(12) United States Patent
(10) Patent N0.:
(45) Date of Patent:
Krzyzanowski et a].
(54)
METHOD OF CONTROLLINGA DEVICE TO
PERFORM AN AC TIVITY-BASED OR AN
EXPERIENCE-BASED OPERATION
(56)
References Cited
5,414,426 A
5/l995 O’Donnell et a1.
5,519,878 A
5/l996 Dolin, Jr.
Justin Flores, Boca Raton, FL (US);
Glenn Harter, Wellington, FL (US)
(Continued)
(73) Assignee: OpenPeak Inc., Boca Raton, FL (US)
(*)
OTHER PUBLICATIONS
Subject to any disclaimer, the term of this
patent is extended or adjusted under 35
Total Remote Software.‘ Grg'?n Technology Downloads, 1 page,
printed from InternetiURLihttp://WWW.grif?ntechn0l0gy.c0m/
software/softwareitotalremote.html, (printed Feb. 23, 2004).
U.S.C. l54(b) by 477 days.
(Continued)
(21) Appl. N0.: 11/042,17s
(22)
Filed:
Primary ExamineriNiketa l Patel
(74) Attorney, Agent, or FirmiSterne, Kessler, Goldstein &
Jan. 26, 2005
(65)
Fox P.L.L.C.
Prior Publication Data
US 2005/0128048 A1
Feb. 23, 2010
U.S. PATENT DOCUMENTS
(75) Inventors: Paul Krzyzanowski, FanWood, NJ (US);
Notice:
US 7,668,990 B2
(57)
Jun. 16, 2005
ABSTRACT
Related US. Application Data
Activity-based or experienced-based operations are created
(63) Continuation-in-part of application No. l0/783,5ll,
and de?ned during an initialization or setup session for one or
more system component, With minimum or no user interac
?led on Feb. 23, 2004, and a continuation-in-part of
tion. The system components include legacy and/or non
legacy devices, Which are con?gured and managed Within a
application No. l0/782,923, ?led on Feb. 23, 2004,
noW Pat. No. 7,024,256, and a continuation-in-part of
controlled environment, such as a home or business peer-to
peer or client-server network. During an initialization or
application No. 10/783,017, ?led on Feb. 23, 2004,
noW Pat. No. 7,129,855, and a continuation-in-part of
application No. l0/387,590, ?led on Mar. 14, 2003.
setup session, relevant device information and device meta
data are collected via a conventional or discovery protocol
(51)
interface. For a legacy device that is not automatically dis
coverable, a user adds the legacy device to the controlled
Int. Cl.
G06F 13/12
G06F 3/00
G06F 15/177
(2006.01)
(2006.01)
(2006.01)
environment by inputting the device information. The device
(52)
US. Cl. ............................. .. 710/72; 710/8; 710/62;
information can also be automatically collected from an
accessible database. Once the device information has been
709/220; 709/221; 709/222
collected, the system components are con?gured to support
(58)
Field of Classi?cation Search ................. ..
one or more activity-based or experienced-based operations.
710/72,
710/8, 62; 709/22(L222
See application ?le for complete search history.
27 Claims, 12 Drawing Sheets
100
,/
106(a)
104(b)
‘
106(b)
(Eg..
Audio/Video
Stream
118
116
Leygucyrevlrces
d ' ’
130(a)
Thermoslu?
114
130(b)
140
US 7,668,990 B2
Page 2
US. PATENT DOCUMENTS
5,537,463 A
5,552,917 A
5570085 A
7/1996 Escobosa et al.
9/ 1996 Darbee e131
10/ 1996 Bertsch
2001/0053274
2001/0054060
2001/0055954
2002/0006788
2002/0019984
5,689,353 A
11/ 1997 Darbee e131
A1
A1
12/2001 Roelofs et al.
12/2001 Fillebrown et al.
A1
A1
A1
12/2001 Cheng
1/2002 Knutsson etal.
2/2002 Rakib
2002/0022991 A1 *
2/2002
5,771,388 A
6/1998 Mondrik et al.
2002/0031120 A1
3/2002 Rakib
5,802,467
5,805,812
5,875,108
5,898,386
5,926,108
A
A
A
A
A
9/1998
9/1998
2/1999
4/1999
7/ 1999
2002/0033760
2002/0034292
2002/0035403
2002/0035404
2002/0035605
A1
A1
A1
A1
A1
3/2002
3/2002
3/2002
3/2002
3/2002
Kobayashi
Tuoriniemietal.
Clark etal.
F1000 etal.
McDowell et al.
5930699
5,953,144
5,956,025
5956487
5959751
5963624
6,005,861
6026150
6,032,202
6,049,711
6,052,750
A
A
A
A
A
A
A
7/ 1999
9/1999
9/1999
9/1999
9/1999
10/ 1999
12/ 1999
2002/0035621
2002/0036795
2002/0068984
2002/0087746
2003/0103088
2003/0105854
2003/0163626
A1
A1
A1
A1
A1
A1
A1
3/2002
3/2002
6/2002
7/2002
6/2003
6/2003
8/2003
Zintel et al.
Fujitani
Alexander et al.
Ludtke etal.
Dresti et al.
Thorsteinsson etal.
Chen et al.
A
A
2/2000
2/2000
4/2000
4/2000
2003/0191826
2003/0204582
2003/0207668
2003/0208595
A1
A1
A1
A1
10/2003
10/2003
11/2003
11/2003
A
A
6085236 A
6,131,028
6148205
6154745
6,167,046
6,170,007
A
A
A
A
B1
5812121118181
Fish @131
Ho?berg e1a1~
Kalhatsu
Wicks er a113113113
Darbee er 31Goulden et 31Venkatraman e1a1~
Darbee er 31Pope
Humpleman
Fran1<e1a1~
Lea @1211
Ben-Yehezkel et 31Lea
7/2000 Lea__
111/2000
11/2000
11/2000
12/2000
1/2001
Wh?mgton
Cotton
Kari er 31Terada e131Venkatraman e1a1~
Sharood et al. ............. .. 705/14
Bellingeretal.
Shimoda etal.
McFarland et al.
Gouge et al.
2003/0220988 A1
11/2003 Hymel
2003/0224735
2003/0224807
2003/0236890
2004/0003051
2004/0003073
12/2003
12/2003
12/2003
1/2004
1/2004
A1
A1
A1
A1
A1
Moursund et al.
Sinha et al.
HurWitZ et al.
KrZyZanoWski et al.
Krzyznowski etal.
6,198,479 B1
3/2001 Hqmpleman e131
2004/0046677 A1
3/2004 Drestietal.
6,199,066 B1
6,199,136 B1
3/2001 Glltho e131
3/2001 Shteyn
2004/0047347 A1
2004/0054789 A1
3/2004 Worry etal.
3/2004 Breh etal.
6,208,341
6,208,866
6 a 218 a 931
6222
530
a
a
3/2001 Van E9 9‘ 91'
2004/0071136 A1
4/2004 Laumen etal.
40001 AS g har et a1‘
40001 Se queira
2004/0098456 A1
2005/0015764 A1
5/2004 Krzyzanowskl et al.
1/2005 Gaur
6,223,348 B1
40001 Hayes et a1‘
2005/0108369 A1
6,243,707
6,243,772
6,259,707
6,266,612
6,275,865
6,282,714
6/2001
6/ 2001
7/2001
7/2001
8/2001
8/2001
B1
B1
B1
B1
B1
B1
B1
B1
B1
B1
6,288,716 B1
6,292, 554
6,317,028
6,317,718
6,321,092
B1
B1
B1
B1
3/2001 Rouhollahzadeh et al.
Humpleman et al.
Ghori et al.
Dara-Abrams et al.
Dussell et 31‘
Zou
Ghoriet a1‘
9/2001 Humpleman etal.
9/2001
11/2001
11/2001
11/2001
Oden et al‘
Valiulis
Fano
Fitch et 31,
.
>1<
5/2005
Sather et al. .............. .. 709/220
OTHER PUBLICATIONS
Total Remote
v2.0 user guide,
29
pages,
printed from
InternetiURLihttp://WWW.grif?ntechnology.com/downloads/
pdf/manuals/TotaliRemoteiuserimanual.pdf, (printed Feb. 23,
2004)
International Search Report for International Application No. PCT/
US03/20189, Issued on Oct. 29, 2003.
“Global Caché GC-100 Network AdapteriControl & Automate
Common Devices OverA Network” GC-100 NetworkAdapter, URL
6,349,352 B1
2/2002 Lea
http://WWWglobalcache.com/products/gc-index.htrnl, Oct. 11, 2004,
6,359,636 B1
3/2002 Schindler et al.
2 pages.
6,363,434 B1
6,388,399 B1
6,405,261 B1
3/2002 Eytchison
5/2002 Eckelet 31,
6/2002 Gaucher
6,463,343
6,496,135
6,512,754
6,563,430
6,587,067
6,597,374
B1
B1
B2
B1
B2
B1
6,600,421 B2
6,642,852 B2
10/2002
12/2002
1/2003
5/2003
7/ 2003
7/2003
7/2003
ll/2003
Emens et al.
Darbee
Feder et al.
Kemink et al.
Darbee et al.
Baker et 31.
Freeman
DreSti et a1.
“CC-100 Key Features and Bene?ts” GC-100 Network Adapter,
URL http://WWWglobalcache.com/products/gc-features.htrnl, Oct.
11, 2006, 2 pages.
“Global Caché GC-100 Network AdapteriControl & Automate
Common Devices Over A Network”, Product Data Sheet, 2 pages,
2004.
“IPL TSFI244, Two Serial, Four Flex I/O, andFourIR PortIPLinkTM
Ethernet Control Interface With IR Learner”, EXtron IPL T SF1244
IP LinkTM Products, URL http://WWW.eXtron.com/product/product.
asp?id:iplts?244&version?5rint, Oct. 11, 2004, 6 pages.
“Creston Introduces RMC2e “ll/lini” Control System ” Home Toys
6,647,426
6,728,784
6,741,853
6,748,343
6,792,469
6,914,551
2001/0000194
B2
B1
B1
B2
B1
B2
A1
ll/2003
4/2004
5/2004
6/2004
9/2004
7/2005
4/2001
Mohammed
Mattaway
Jiang et a1~
AleXaIldeT e1 a1~
Callahan et al.
Vidal
Sequeira
New Release, URL http://WWW.hometoys.con1/releases/mar03/
crestoni02.htrnl, Oct. 14, 2004, 2 pages.
“Troy Serial Server,” URL, http://WWW.troygroup.com/Connectiv
ity/products/SerialDeviceConnectivity/SerialServers.asp,
doWn
loaded Jul, 9, 2004, 2 pages,
“Troy Serial Servers,” URL, http://WWW.troygroup.com/Connectiv
ity/products/Datasheets/
2001/0018663
2001/0036192
2001/0041561
200 1/ 0049275
2001/0049846
A1
A1
A1
A1
A1
8/2001
11/2001
ll/200l
12/ 2001
12/2001
Dussell e1 31~
Chiles et al.
Venflllett et a1.
Pierry et al.
GuZZi et al.
Troy%20Serial%20Server%20data%20sheet.pdf, downloaded Jul.
9, 2004, 1 page,
“EtherWindTM 802.11bTM Wireless Print Server”, URL http://WWW.
troygroup.com/Wireless/documents/datasheets/
TROYU%20EtherWind.pdf, Nov. 19, 2003, 2 pages.
US 7,668,990 B2
Page 3
“EtherWindTM IEEE 802.11b Wireless Print Server”, URL http://
“Harmony RemoteiFeatures”, URL http://wwwharmonyremote.
www.troygroup.con1/wireless/products/wireless/etherwind.asp,
com/productifeatureshtrn, downloaded from the World Wide Web
Nov. 19, 2003, 3 pages.
“Troy Wireless Introduces EtherWind-Plus 802.11b Wireless OEM
Board-Level Product; New Product Enables Wireless and Ethernet
Connectivity on OEM Devices”, URL http://www.troygroup.con1/
wireless/company/news/08l302.asp, Nov. 19, 2003, 2 pages.
“EtherWind-PlusTM 802.11b OEM Module”, URL http://www.
on Feb. 9, 2004, 5 pages.
troygroup.com/wireless/documents/datasheets/
TROY%20EtherWind%20Plus%20data%20sheet.pdf,
Nov.
19,
2003, 2 pages.
“EtherWind-PlusTM IEEE 802.11b OEM Connectivity Module”,
URL
http://www.troygroup.con1/wireless/products/wireless/
etherwindplus.asp, Nov. 19, 2003, 2 pages.
“Speci?cations for Harmony Remote”, URL http://www.
harmonyremote.com/productispeci?cations.htm, downloaded from
the World Wide Web on Feb. 9, 2004, 2 pages.
“HarmonyRemoteiFrequentb/Asked Questions”, URL http://www.
harmonyremotecom/faqhtrn, downloaded from the World Wide
Web on Feb. 9, 2004, 2 pages.
“Harmony
Remote
SST-659”,
published
at
http://www.
harmonyremotecom, downloaded from the World Wide Web on Feb.
9,2004, 1 page.
“Harmony
Remote
SST-748”,
published
at
http://www.
harmonyremotecom, downloaded from the World Wide Web on Feb.
9,2004, 1 page.
“Pronto RF Extender, RFX 6000, Pronto accessory for operating
“Harmony
devices inside a closed cabinet, closet or even in another room”, URL
harmonyremotecom, downloaded from the World Wide Web on Feb.
9,2004, 1 page.
http://wwwpronto.philips.com/index.cfm?id:577f, Nov. 19,2003, 2
Remote
SS T- 768t”,
published
at
http://www.
“Pronto RF Extender, RFX 6000, RF extender for TSU6000 &
“Harmony Remote SST-659 User Manual”, dated Oct. 16, 2003,
published at http://www.harmonyremote.com, downloaded from the
TSU3000”, URL http://wwwremotecontrol.philips.con1/library/
World Wide Web on Feb. 9, 2004, 20 pages.
documents/SPECiRFX6000.pdf, Nov. 19, 2003, 2 pages.
“Harmony Remote SST-768 User Manual”, dated Mar. 25, 2003,
published at http://www.harmonyremote.com, downloaded from the
pages.
“Harmony RemoteiOverview”, URL http://wwwharmonyremote.
com/productioverviewhtrn, downloaded from the World Wide Web
World Wide Web on Feb. 9, 2004, 37 pages.
on Feb. 9, 2004, 2 pages.
* cited by examiner
US. Patent
Feb. 23, 2010
Sheet 1 0f 12
US 7,668,990 B2
6;W..."
.,
A302
EomEw.,
91N:Ao.vomw
.{E tumoctwrc.
362
US. Patent
Feb. 23, 2010
Sheet 3 0f 12
US 7,668,990 B2
300
,1
303
Add or Remove
Rooms
1
306
Assign WiFi/IR
/
Controllers to Rooms
l
309
/
Assign Detected UPnP/
Networked Devices to Rooms
l
312
Add Legacy Devices
/
to Rooms
i
Con?gure Devices
,
395
FIG. 3
US. Patent
Feb. 23, 2010
Sheet 4 0f 12
wow
fl2:,1.LI
W$Q5E0:P
.5
.
US 7,668,990 B2
US. Patent
Feb. 23, 2010
Sheet 5 0f 12
US 7,668,990 B2
wow
.UEaw
wow
i@5E20:5 .
wow
wow
US. Patent
Feb. 23, 2010
Sheet 8 0f 12
US 7,668,990 B2
ooh
F03 %;
@gwmcEo:.
*025;0E2aEL?03%0%252
9Mmw385E2:oN08I5»o?w
.OEh
US. Patent
cow
Feb. 23, 2010
Sheet 9 0f 12
US 7,668,990 B2
Now
w.9“
k
,
..
Q52%9.0.2,
,mtmhmw
96W.
39:, 31 :
38$8Y-
3%$0
US. Patent
Feb. 23, 2010
Sheet 12 0f 12
US 7,668,990 B2
/\/
EHO
1100
Processor 1 104
Main Memory 1108
Display Interface 1102
Display 1130
Secondary Memory 1110
Communication
Infrastructu re
Hard Disk Drive
1112
1 106
Removable Storage
Removable
Drive 1114
Storage Unit 1118
Interface 1120
Communications
Interface 1124
Removable
Storage Unit 1 122
X
Communications Path
1126
FIG. 11
US 7,668,990 B2
1
2
METHOD OF CONTROLLINGA DEVICE TO
PERFORM AN AC TIVITY-BASED OR AN
EXPERIENCE-BASED OPERATION
memory. HoWever, the user cannot inform the RCU of critical
operational information of the controllable devices (e.g., tele
vision, DVD player, etc.).
Such critical operational information includes the input
state (i.e., What signals are coming into What input).
CROSS-REFERENCE TO RELATED
APPLICATIONS
For example, if a television is set to the tuner input (instead
of the input for receiving DVD data) When a RCU is operated
lian et al., entitled “User Interface for Multi-Device Control,”
to play a DVD installed in a DVD player, the television Will
display the broadcast channel to Which it is tuned at the time,
and not the desired DVD. The user must physically select the
incorporated herein by reference in its entirety.
proper input by cycling through the available inputs using the
This application is a continuation-in-part of US. patent
application Ser. No. 10/783,511, ?led Feb. 23, 2004, by Kil
input selection button. The lack of input state information also
precludes the user from sWitching betWeen different activities
This application is also a continuation-in-part of US.
patent application Ser. No. 10/782,923, ?led Feb. 23, 2004, by
KrZyZanoWski et al., entitled “Method, System, and Com
puter Program Product for Automatically Managing Compo
(e.g., Watching DVD, Watching broadcast TV, Watching
cable) Without also physically changing from the current
input state to the neW state, thereby placing the television in
the proper input state.
nents Within a Controlled Environment,” incorporated herein
by reference in its entirety.
Although the combination of improved computer process
This application is also a continuation-in-part of US.
patent application Ser. No. 10/783,017, ?led Feb. 23, 2004, by
KrZyZanoWski et al., entitled “Device Control System,
Method, and Apparatus,” incorporated herein by reference in
its entirety.
20
devices connected to a residential netWork and facilitate the
This application is also a continuation-in-part of US.
patent application Ser. No. 10/387,590, ?led Mar. 14, 2003,
by KrZyZanoWski et al., “Legacy Device Bridge for Residen
tial or Non-Residential Networks,” incorporated herein by
reference in its entirety.
BACKGROUND OF THE INVENTION
operations of these devices With minimal user intervention.
25
SUMMARY OF THE INVENTION
A method, system, and computer program product are
provided to con?gure and manage a plurality of system com
30
sumer electronics, and more speci?cally, to a process for
35
ling consumer electronics.
2. Related Art
computer netWork can be time extensive and expensive to
deploy. Additionally, there is no easy method to integrate, for
example, home computer netWorks With other residential
devices, such as televisions, stereos, DVD players, and other
home electronics. Being able to ef?ciently distribute digital
audio/video (AV) data among personal computers and other
AV devices (such as, televisions, DVD players, PVRs, etc.) is
ity-based or experienced based operations include favorite
40
trols, control macros (including context-based control mac
ros), or the like.
45
dards and/or formats.
50
control netWorked residential devices. Although a remote
control unit (RCU) can be trained to send signals to compo
nents of an entertainment center (such as, a television, stereo,
and DVD player), a conventional RCU cannot communicate
and control multiple personal computers and other analog
settings, playlists, security protocols, context-sensitive
operations, user-speci?c operations, folloW-me system con
complicated by differing and evolving communications stan
Another signi?cant challenge is being able to effectively
devices, and the controlled environment includes a peer-to
peer communications system and/or a client-server netWork.
In an embodiment of the present invention, “activity
based” or “experienced-based” operations are created and
de?ned during an initialiZation or setup session for a system
component, With minimum or no user interaction. The activ
The installation of computer netWorks (for homes and busi
nesses) presents several challenges. For example, installing a
ponents Within a controlled environment, such as a home,
business, school, etc, as Well as its surrounding areas. The
system components include legacy and/or non-legacy
1. Field of the Invention
The present invention relates generally to control of con
creating activity or experience-based operations for control
ing capabilities and global access to information has resulted
in signi?cant advancements in the information processing
age, there exists a need for a simple, inexpensive, yet versatile
system that can integrate the functions of multiple residential
For non-legacy devices, relevant information about the
system component is collected via a discovery protocol inter
face. For legacy devices, a variety of system interfaces are
provided for receiving the relevant information.
In an embodiment, the system components (both legacy
and non-legacy devices) are added, removed, or con?gured
via a setup WiZard. During this initialiZation or setup process,
relevant device information (e.g., device type, make, model,
region assignment, etc.) is automatically discovered or added
by user input. For a legacy device that is not automatically
discoverable, the user adds the legacy device by identifying
55
and/or digital devices at a residence. In addition, a conven
the device (by device type, make, model, etc.) and assigning
tional RCU cannot communicate and control multiple resi
the device to a particular room/region.
The initialiZation or setup process is also designed to
dential devices Without user intervention. Even With a con
acquire “device metadata,” Which includes basic operational
ventional RCU, a user must manually activate a hard key to
send a command (in the form of infrared signals) to a con
information or idiosyncrasies about each neW device (such as,
60
tuning rules, types and order of inputs, etc.). The concept of
trollable consumer electronic (CE) device having direct line
metadata is further described in the application entitled
of-sight connectivity With the RCU.
“Device Control System, Method, and Apparatus” (US.
patent application Ser. No. 10/783,017). The present inven
A conventional RCU suffers from other serious de?cien
cies that limit its usefulness Without constant user interven
tion. For instance in the “set up” process of a conventional
RCU, the user can program the device by assigning a particu
lar key With a series of key strokes that are stored in the RCU’ s
tion includes mechanisms for optimiZing the collection of
65
device metadata With little or no human intervention. In an
embodiment, a series of questions or informational prompts
are dynamically optimiZed to avoid seeking irrelevant or
US 7,668,990 B2
3
4
unnecessary information. In an embodiment, device metadata
is contained in an accessible database. As such, the setup
process can be tailored and streamlined based on the type and
model number of the system component being added or con
protocol interface, non-legacy devices are specially adapted
for discovery and control by other devices.A “legacy device,”
?gured. The setup process is dynamically tailored to seek
only that Which is speci?c to and necessary to control the
discovery protocol interface. Lacking a discovery protocol
interface, legacy devices, therefore, are not conventionally
adapted for discovery and control by other devices.
According to embodiments of the present invention during
on the contrary, is de?ned herein to include any type of device
or peripheral that is not capable of communicating via a
particular system component in light of its user-speci?c con
?guration.
Once the device information (including device metadata)
has been collected, the system components (e.g., the assigned
the initialiZation or setup phase, the system component is
con?gured to perform one or more activity-based or experi
legacy and non-legacy devices) are con?gured to support one
enced-based operations With minimum or no user interaction.
or more activity-based or experienced-based operations.
For non-legacy devices, relevant information about the sys
tem component is collected via a discovery protocol inter
face. For legacy devices, a variety of system interfaces are
provided for receiving the relevant information, as described
in greater detail beloW.
In an embodiment, the activity-based or experience-based
BRIEF DESCRIPTION OF THE
DRAWINGS/FIGURES
The accompanying draWings, Which are incorporated
herein and form part of the speci?cation, illustrate the present
invention and, together With the description, further serve to
explain the principles of the invention and to enable one
operations include one or more “favorite” settings for the
the draWings, generally, like reference numbers indicate iden
system component. For example, a favorite setting can be
established for television programming, audio/video record
ings, room temperature, hot tub controls, clock alarms, light/
dimmer settings, Web sites, neWs broadcasts, ?nancial chan
tical or functionally or structurally similar elements. Addi
nels, or the like.
20
skilled in the pertinent art(s) to make and use the invention. In
tionally, generally, the leftmost digit(s) of a reference number
identi?es the draWing in Which the reference number ?rst
appears.
FIG. 1 illustrates a residential controlled system according
to an embodiment of the present invention.
FIG. 2 illustrates a residential controlled system according
to another embodiment of the present invention.
In another embodiment, the activity-based or experience
25
30
FIG. 3 illustrates an operational ?oW for initialiZing system
based operations include one or more “security” protocols for
FIGS. 4a-4b illustrate a room control screen according to
35
can be prevented from accessing designated television chan
nels, Web sites, areas (such as, lockable rooms, draWers, safes,
40
In another embodiment, the activity-based or experience
nent can have one or more con?gurations based on a particu
lar context. Depending on the type of system component, a
45
controls, or the like. Another con?guration can be established
50
coffee maker timer controls, or the like. A con?guration can
timer controls, lighting, neWs broadcasts, or the like.
The present invention describes methodologies and/or
techniques for con?guring and managing a system compo
60
In another embodiment, the activity-based or experience
based operations are “user-speci?c”. For example, a user
speci?c con?guration can be established With security con
trols set to block certain televisions programming, Web sites,
audio recordings, or the like When in the company of minors.
HoWever, When in the company of adults, a con?guration can
be established With a loWer security setting.
In another embodiment, the activity-based or experience
based operations include “folloW-me” system controls. For
instance, “folloW-me video” is implemented to transfer a
implemented With a peer-to-peer communications system
and/ or a client-server netWork. The system component
includes legacy as Well as non-legacy devices. A “non
sal Plug and Play (UPnPTM) Forum). Having a discovery
for home of?ce activities, Which includes, Without limitation,
presets for a baby monitor, a playlist of classical recordings,
be established for morning rituals, Which includes, Without
limitation, presets for alarm/snooze controls, coffee maker
I. OvervieW
ery protocol interface (e. g., interfaces de?ned by the Univer
con?guration can be established for evening entertainment,
Which includes, Without limitation, presets for lighting,
audio/video presentations, security access Warnings, hot tub
55
legacy” device is de?ned herein to include any type of device
or peripheral that is capable of communicating via a discov
etc.), or the like.
based operations are “context-sensitive”. The system compo
DETAILED DESCRIPTION OF THE INVENTION
nent of a controlled environment (such as, a residence, busi
ness, school, or the like). The controlled environment can be
the system component. For example, the system component
can be con?gured to block certain content from being
accessed by a designated user. Non-adult users, for instance,
device metadata according to another embodiment of the
present invention.
FIG. 11 is an example computer system useful for imple
menting the present invention.
recordings can be prepared and/ or stored for playback on, for
example, a television, monitor, or audio client. A sequence of
graphic images or photographs can be prepared and/or stored
for playback on, for example, a monitor or computer client. A
playlist of other forms of media and/or multimedia can also be
created according to embodiments of the present invention.
In another embodiment, the activity-based or experience
components in a controlled environment according to an
embodiment of the present invention.
an embodiment of the present invention.
FIG. 5 illustrates a Wireless controller device control
screen according to an embodiment of the present invention.
FIG. 6 illustrates a non-legacy device control screen
according to an embodiment of the present invention.
FIG. 7 illustrates a legacy device control screen according
to an embodiment of the present invention.
FIG. 8 illustrates a legacy device control screen according
to another embodiment of the present invention.
FIG. 9 illustrates a device control screen for specifying
device metadata according to an embodiment of the present
invention.
FIG. 10 illustrates a device control screen for specifying
based operations include one or more “playlists” for the sys
tem component. For example, a series of video or audio
65
selected video production to various displays throughout the
controlled environment. As a user migrates from room to
room Within the controlled environment, the user, or a system
US 7,668,990 B2
5
6
component in communications With the user, is tracked to
determine the user’s current location, and a display (e.g.,
monitor or television) positioned in the current location Will
Similarly, the activity-based or experience-based operations
herein by reference in its entirety; and in US. Pat. No. 6,792,
323 to KrZyZanoWski et al., Which is incorporated herein by
reference in its entirety
As described in greater detail beloW, activity-based or
experienced-based operations are created and de?ned during
can include “folloW-me audio,” “folloW-me lighting,” or the
like.
function and/or operation of system components, including
automatically start to present a pre-selected video production.
an initialiZation or setup session, and are executed With the
In another embodiment, the activity-based or experience
consumer electronic (CE) devices. The activity-based or
based operations are de?ned by one or more control macros.
experienced-based operations support the control of multiple
A control macro includes a set of commands that, When
devices of the same type (e. g., tWo DVD players). The activ
executed, enables a controller to control multiple operations
ity-based or experienced-based operations also support the
and/or functions of one or more system components. In an
segmentation of devices among multiple rooms (e.g., a tele
embodiment, the set of commands are stored in a central
location and are associated With a generic command. When
vision in a living room versus a television in a bedroom); the
control of non-IR, netWorked devices (such as devices having
a discovery protocol interface); the control of devices beyond
line-of-sight (via a Wireless netWork); and setup directly on a
remote control device (as opposed to a ?xed stationary inter
activated, the generic command retrieves and transmits the set
of commands to the appropriate system component(s) for
execution. In other Words, the present invention enables a
remote or portable controller to transmit a single high-level
face).
request to a centraliZed command center (Which can be
coupled to another computer). The centraliZed command cen
20
ter interprets the single request and ?nds the corresponding
sequence of commands that needs to be transmitted to the
II. Example Controlled Environment
The present invention can be implemented in any type of
controlled environment, such as a residence, business, school,
appropriate system component(s).
or the like. A residential controlled environment includes the
In an embodiment, the control macro is de?ned to execute
automatically upon the occurrence of a commencement
parameter. The commencement parameter can be a pre
con?nes of a home, apartment, mobile home, houseboat, or
other types of residences. HoWever in embodiments, a resi
dential environment includes the surrounding area of the resi
dence, as Well as any shelters, constructs, improvements, or
25
de?ned time, date, state, event, or the like, or any combination
thereof. For example, a pre-de?ned sequence of control com
the like, Within a designated perimeter.
In other embodiments, the present invention is imple
mands can be executed upon the occurrence of a pre-set time
(i.e., clock based) and/ or date (i.e., calendar based) automati
30
mented in a non-residential environment. A non-residential
cally (i.e., Without direct user intervention). A user can desire
to have the bedroom television automatically turned on and
environment includes, but is not limited to, an of?ce complex,
tuned to a particular channel at 6:30 am as part of his or her
tainment arena, school or university, health care facility,
suite of small of?ces, production studio, Warehouse, enter
hotel, vacation resort, aircraft, ship, automobile, or the like. In
“Wake up” routine. In another example, a pre-de?ned
ticular system or component state. A control macro can be
embodiments, the controlled environment for a non-residen
tial embodiment includes not only the actual con?nes of the
aforementioned structures but also their surroundings Within
automatically executed When, for example, a television in the
a designated perimeter.
sequence of control commands can be executed upon the
35
occurrence of a particular event or the occurrence of a par
Examples of a controlled environment are described in the
family room has been poWered on.
In an embodiment, the control macro is de?ned to be auto
40
application entitled “Method, System, and Computer Pro
matically particulariZed for application in a speci?c context.
gram Product for Managing Controlled Residential or Non
Herein referred to as a “context-based control macro”, a con
Residential Environments” (U .S. patent application Ser. No.
trol macro can be particulariZed for use in any context (e.g.,
10/382,897), and the application entitled “Method, System,
and Computer Program Product for Managing Controlled
any region, or any device of comparable type). In other Words,
a speci?c system component or combination of components
45
Residential or Non-Residential Environments,” (U .S. patent
application Ser. No. 10/180,500), both of Which are incorpo
rated herein by reference in their entireties. As described in
these applications, various methods and systems can be pro
vided to manage the distribution of information (including
50
video, audio, voice, text, graphics, control messages, etc.) to
is not identi?ed in the control macro. A context-based control
macro is de?ned by specifying the “type” of system compo
nent that is required to perform the activity-based or experi
enced-based operation. Upon activation, the control macro is
interpreted to select the appropriate system component(s). In
an embodiment, the system component is selected by deter
other devices and/or applications Within the controlled envi
ronment. Such devices and/or applications include, but are
not limited to, communications equipment (such as, tele
mining the current location of the user (or a system compo
nent in communications With the user). In another embodi
ment, the user interacts With a remote orportable controller to
specify the appropriate system component(s) at the time of
55
activating the context-based control macro.
The above-mentioned activity-based or experience-based
operations have been provided by Way of example and not
limitation. Other types of activity-based or experienced
based operations canbe used and are intended to be Within the
phones, intercoms, etc.), entertainment systems (such as,
televisions, CD/DVD players, gaming applications, stereos,
etc.), monitoring systems (such as, security cameras, baby
monitors, etc.), safety/security systems (such as, ?re alarms,
sprinkler systems, locks on doors or WindoWs, etc.), personal
60
computers (such as, desktops, notebooks, notepads, personal
digital assistants, etc.), cooking appliances (such as, ovens,
scope of the present invention. Additional examples of activ
coffee makers, electrical food/beverage Warmers, etc.), com
ity-based or experience-based operations (including further
fort systems (such as, heating and air conditioning (HVAC),
descriptions of the above-mentioned operations) are
humidi?ers, dehumidi?ers, air puri?ers, light sWitches, light
described in the application entitled “Method, System, and
Computer Program Product for Automatically Managing
Components Within a Controlled Environment” (U .S. patent
application Ser. No. 10/782,923), Which is incorporated
65
dimmers, etc.), poWer outlets, poWer supplies, or the like.
An example of such controlled environments is shoWn in
FIG. 1, Which illustrates a residential controlled system 100
according to an embodiment of the present invention. System
US 7,668,990 B2
7
8
100 includes a communications network 180 that intercon
Wireless local area netWork (LAN). Thus, communications
netWork 180 includes Wired, Wireless, or both transmission
nects a plurality of system components. The system compo
nents include a positioning unit 102, tWo televisions 104
(shoWn as television 104(a) and 104(b)), tWo computer cli
ents 106 (shoWn as computer client 106(a) and computer
client 106(b)), a portable controller 108, a lighting device
media, including satellite, terrestrial (e. g., ?ber optic, copper,
UTP, STP, coaxial, hybrid ?ber-coaxial (HFC), or the like),
radio, microWave, free-space optics, and/or any other form or
method of transmission.
As shoWn in FIG. 1, communications netWork 180 sup
110, a thermostat 112 for a HVAC system, a tuner 114, a
ports peer-to-peer communications. HoWever, the present
media player 116, a cable box 118, a digital satellite system
(DSS) box 120, and a Wireless lighting control interface (such
invention also supports client-server netWorks. FIG. 2 illus
trates another embodiment of residential controlled system
as a Lutron or X10 transceiver) 140. Other devices and/or
applications can also be included as system components.
100 that includes at least one server 208. Server 208 polices
Positioning unit 102 designates spatial locations Within the
all tra?ic among the other system components. The exchange
residence that serves as the hosting environment for system
of information among the system components is routed or
100. Positioning unit 102 is coupled to the other system
components (e.g., portable controller 108) via a Wired and/or
Wireless interface. Positioning unit 102 is operable to desig
otherWise controlled via server 208. As such, server 208
interacts With other system components to directly or indi
rectly distribute data (including audio and/or video), voice,
and/or control messages over communications netWork 180.
In an embodiment, server 208 commands and controls the
operations and/or functions of one or more of the other sys
nate a ?oor or room Within the residence. Positioning unit 1 02
is also operable to designate a speci?c location or region
Within a ?oor or room. Moreover, positioning unit 102 can be
situated outside of the residence to thereby, designate external
20
?le sharing, Internet access, and security. According to
Computer client 106 includes a Wired and/or Wireless per
sonal computer, personal digital assistant (PDA), enhanced
telephone, personal television, or other data processing
device linked to communications netWork 180. As a personal
computer, computer client 106 can be a desktop, notebook,
notepad, or the like.
Portable controller 108 is a Wired and/or Wireless data
processing device that enables a user to interact With, send
control messages to, and/ or manage the distribution of infor
tems components. The functions managed by server 208
include video serving, audio serving, telephony, messaging,
areas of the residence.
embodiments of the present invention, a user operates por
table controller 108 to establish or recon?gure these functions
25
and/or receive media from server 208 or other systems com
ponents or indirectly from the system components via server
208.
In an embodiment, portable controller 108 includes several
functions of server 208, as described herein, and manages the
30
mation (including audio, video, voice, and other data) among
distribution of information (including audio, video, voice,
and other data) among the other system components. In
the other system components. Portable controller 108 can be
another embodiment supporting peer-to-peer communica
a portable version of the devices listed as computer client 106.
For example, portable controller 108 can be a personal note
tions, the system components exchange audio, video, voice,
book or notepad computer, PDA, enhanced telephone, or
other data, and/or control messages directly With each other
35
other device linked to communications netWork 180 and
including a display With the ability to interact With the other
system components. Hence, portable controller 108 enables a
user to remotely control the operations of various components
of system 100. In an embodiment, the display for portable
controller 108 is capable of receiving video and/or audio from
the other system components. In an embodiment, portable
In an embodiment using a Wired transmission medium,
communications netWork 180 is an Ethernet LAN having a
40
CAT-5 cable, or the like, distributed to each system compo
nent, such as television 104, media player 116, etc. The sys
tem component includes an audio/video (AV) connector that
is responsive to receive the cable. In an embodiment, com
munications netWork 180 includes a telephone line and/or
controller 108 includes a ?ash ROM that enables Wireless
doWnloads and/or uploads.
Television 104 is a conventional television. In an embodi
and Without being centrally managed.
45
poWerline (such as the communications technologies made
available from the Home Phone Networking Alliance (Home
PNA) or the like).
ment, television 104 is enhanced to support interactive and/or
personal services. Personal services include virtual record
In an embodiment using a Wireless transmission medium,
communications netWork 180 supports the IEEE standard
ing, programming, pausing/reWinding live broadcasts, or the
802.11(a), Which speci?es a Wireless Ethernet protocol for
large-siZed video. In another Wireless embodiment, commu
nications netWork 180 supports the IEEE standard 802.1 1 (b),
Which speci?es a Wireless Ethernet protocol for small-siZe
video. In another Wireless embodiment, communications net
Work 180 supports the IEEE standard 802.11(g). In another
like. For example, television 104 can be a personal television
enhanced to support the MSN® TV service, hosted by
50
WebTV NetWorks, Inc. (MountainVieW, Calif.), that supports
the WEBTV® services available from Microsoft Corporation
(Redmond, Wash). As shoWn, television 104 can be con
nected to cable set-top box 118, DSS set-top box 120, and/or
media player 116 (e. g., PVR, VCR, or DVD player).
55
Wireless interfaces With system 100.
As shoWn in FIG. 1 and FIG. 2, communications netWork
The aforementioned system components are not intended
to be exhaustive. Other devices (including appliances), appli
cations, and/or the like can be implemented, including, but
180 includes a Wireless netWork access point 182, such as
not limited to, a refrigerator, stove, microWave, toaster, cof
fee-maker, alarm clock, humidi?ers, sprinkler system, light
60
ing, light dimmers, etc. In an embodiment, portable controller
108 controls the operations and/or functions of such compo
nents, such as on/off, timers, modulation (e.g., oven tempera
tures, etc.), pause, snooZe, etc.
As discussed, communications netWork 180 provides a
transmission medium for communicating among the system
components. Communications netWork 180 is a Wired and/or
embodiment, the BLUETOOTHTM Wireless technology (de
veloped by Bluetooth SIG, Inc.) is used to support short-range
those available from Linksys, a division of Cisco Systems,
Inc. (San Jose, Calif.), as a platform for the system compo
nents. A Wireless access point 182 provides a central point for
connectivity in a Wireless netWork and alWays-on connectiv
ity necessary for tracking states of the system components.
65
Additionally, a Wireless access point 182 can provide a con
nection point betWeen a Wired and Wireless netWork. In an
embodiment using a server, as discussed above, a Wireless
access point 182 serves as a platform for server 208. In alter
US 7,668,990 B2
10
native server-based embodiments of the present invention,
Co. Ltd. of Osaka, Japan, employs a proprietary protocol for
server 208 can actually be located on a number of different
discovery and control of that device in a network.
device platforms in addition to a wireless access point (such
In an embodiment of the present invention, one or more
as, wireless access point 182), including a bridge device (such
as, bridges 130(a)-130(e)), a personal digital assistant, a per
bridges 130(a)-130(e) act as a plug and play proxy for a piece
of legacy equipment and is herein referred to as a “plug-and
sonal computer, or the like.
play-enhanced” bridge 130(a)-130(e). The “plug-and-play
enhanced” bridge 130(a)-130(e), therefore, includes a virtual
System 100 also includes a plurality of infrared/ serial
plug and play protocol interface that enables the plug-and
play-enhanced bridge 130(a)-130(e) to advertise itself to
bridges 130(a)-130(d), which comply with the IEEE 802.11
(b) standard for wireless communications. Each infrared/se
other network entities as a conventional plug and play device,
exhibiting attributes similar to the legacy device that is com
rial bridge 130 interacts with one or more components. As
shown, infrared/ serial bridge 130(a) interacts with television
104(a), cable box 118, and media player 116. Infrared/serial
bridge 130(b) interacts with tuner 114 or any type of propri
municatively connected to it. When plug and play commands
arrive from the network (e.g., network 180), plug-and-play
enhanced bridge 130(a)-130(e) translates them into an appro
etary device that rely on infrared/ serial communications pro
priate set of commands for controlling the legacy device, such
tocols as would be apparent to one skilled in the relevant
as IR commands or serial commands. Note that although this
art(s). Infrared/ serial bridge 130(c) interacts with thermostat
112. Infrared/ serial bridge 130(d) interacts with lighting
device 110. A wireless-Ethemet bridge 130(e) interacts with
television 104(1)) and a DSS box 120. Wireless-Ethernet
bridge 130(e) can support any IP addressable device. As such,
television 104(1)) and DSS box 120 are “next generation” plug
and play devices that have IP addresses.
Therefore, the present invention can integrate legacy
devices (e.g., CE devices that rely on infrared/ serial commu
nications protocols), as well as devices having a discovery
20
oped, in part, by Apple Computer, Inc. of Cupertino, Calif.
25
embodiment, bridges 130(a)-130(e) are enhanced with a vir
Forum) as system components. An example of a controlled
environment implementing an IEEE 802.1 1(b) infrared/ serial
tual plug and play protocol interface to enable “discovery and
control” of legacy system components within system 100. It
30
Bridge for Residential or Non-Residential Networks” (U.S.
patent application Ser. No. 10/387,590), which is incorpo
130(a)-130(e) can be modi?ed to include a virtual plug and
play protocol interface that enables a legacy device, such as a
conventional CE device, to be advertised to and controlled by
non-legacy system components as if the legacy device were
embodiments, bridges 130(a)-130(e) support other modes of
wireless communications (e.g., RF, WiFi, BLUETOOTHTM,
etc.) that enable discovery and/or control of legacy system
35
is modi?ed to enable discovery and/ or control of legacy sys
with other system components of system 100. In other
embodiments, a bridge 130(a)-130(e) can support a wired
40
interface to a legacy system component and a wireless inter
face to another system components, a wireless interface to a
legacy system component and a wired interface to other sys
tem components, wireless interfaces to both the legacy sys
tem component and other system components, and/ or wired
45
interfaces to both the legacy system component and other
compliant with, for example, a packet-based discovery and
Internet and Web protocols to enable devices such as personal
system components.
computers, peripherals, intelligent appliances, and wireless
III. Setup Optimization
devices to be connected to a controlled environment (such as,
system 100) and to become automatically aware of each
other. In accordance with a plug and play protocol (such as the
UPnPTM Forum), when a user plugs a non-legacy device into
a controlled environment (e.g., system 100), the device con
?gures itself, acquires a TCP/IP address, and uses a discovery
protocol based on the Hypertext Transfer Protocol (HTTP) to
announce its presence to other devices on within the con
components. In another embodiment, a bridge 130(a)-130(e)
tem components over a wired interface (e. g., serial or parallel)
control protocol, such as those de?ned by the UPnPTM Forum.
As would be appreciated by one skilled in the relevant art(s),
the UPnPTM Forum is an open industry standard that uses
should be noted that the wireless aspects of the present inven
tion is not restricted to infrared communications. In other
rated herein by reference in its entirety.
In an embodiment of the present invention, a “virtual plug
and play protocol interface” enables a legacy device to func
tion similarly to a device having a conventional “plug and
play” protocol interface. For example, one or more bridges
Therefore in an embodiment, bridges 130(a)-130(e) are
provided to “control” legacy system components. In another
protocol interface (e.g., devices de?ned by the UPNPTM
bridge is described in the application entitled “Legacy Device
embodiment is described with respect to plug and play virtu
alization, the present invention is not limited to plug and play
virtualization, but also encompasses legacy device virtualiza
tion in accordance with other packet-based discovery and
control protocols, including but not limited to JiniTM, devel
oped by Sun Microsystems of Santa Clara, Calif., or the
RendezvousTM protocol, which is an open protocol devel
50
In an embodiment of the present invention, activity-based
or experienced-based operations are created and de?ned dur
ing an initialization or setup session with minimum or no user
interaction, and are executed with the function and/ or opera
55
tion of various system components. This can be explained
with reference to FIG. 3, where ?owchart 300 represents the
general operational ?ow of an embodiment of the present
trolled environment (e. g., system 100). Other plug-and-play
invention for initializing or setting-up the system components
compliant network entities can then negotiate a communica
in a controlled environment, such as system 100. More spe
ci?cally, ?owchart 300 shows an example of a control ?ow for
tions protocol with the device, determine its capabilities, and
control it. Note that although this embodiment is described
with respect to the UPnP protocol, the present invention is not
limited to the UPnP protocol, but also encompasses other
standards-based discovery protocols, such as the Rendez
establishing activity-based or experienced-based operations
60
?ow is being described with reference to a “residential” con
trolled environment, it should be understood that ?owchart
300 can also be expanded to support non-residential environ
vousTM protocol, developed, in part, by Apple Computer, Inc.
ments.
of Cupertino, Calif. The present invention also encompasses
proprietary discovery protocols designed for speci?c network
devices. For example, the Panasonic® KX-HCM250 Wire
less Network Camera, sold by Matsushita Electric Industrial
during the initialization or setup phase. Although the control
65
The control ?ow of ?owchart 300 begins at step 301 and
passes immediately to step 303. At step 303, a room or region
within the controlled environment (e.g., system 100) is des