JP4434618B2 - Display device - Google Patents

Description

【００３５】
ステップＳ１及びＳ２の処理を行う際は、図１０に示すように、各回の階調値を順に加算する逐次加算を行い、Ｎ回目まで逐次加算を行った後に、Ｎで割ればよい。逐次加算の過程で既に加算済みとなった撮像データは保持しておく必要がない。
【００３６】
図１０のような逐次加算を行う場合、フレームメモリ１６は２回分程度の撮像データを格納できる容量があればよく、メモリ容量を削減できる。
【００３７】
次に、むらパターンの減算処理を行う（ステップＳ３）。次に、ホワイトバランス調整や欠陥補正などを行う（ステップＳ４）。
【００３８】
図１１は、LCD基板１上の信号線駆動回路２２、走査線駆動回路２３、画像取込センサ制御回路２４及び信号処理出力回路２５と、LCDC２と、ベースバンドLSI３との間の信号のやり取りを示す図である。
【００３９】
図１２はガラス基板の詳細構成を示すブロック図である。本実施形態の画素アレイ部２１は、水平方向320画素×垂直方向240画素の表示解像度を有する。バックライトを赤色、緑色及び青色で順繰りに発光させる、いわゆるフィールドシーケンシャル駆動を行うものである。フィールドシーケンシャル駆動ではバックライトの発光色は赤色、緑色及び青色のほか、白色に点灯することもある。画素はそれぞれごとに信号線及び走査線等が設けられる。信号線の総数は、320本で、走査線の総数は240本である。
【００４０】
走査線駆動回路２３は、240段のシフトレジスタ７１と、３選択デコーダ７２と、レベルシフタ（L/S）７３と、マルチプレクサ（MUX）７４と、バッファ７５とを有する。
【００４１】
信号処理出力回路２５は、320個のプリチャージ回路７６と、4選択デコーダ７７と、10段毎にデータバスが接続されたの合計80段のシフトレジスタ７８と、８個の出力バッファ７９とを有する。
【００４２】
図１３は本実施形態の表示装置の動作を説明する図、図１４は通常表示時のタイミング図、図１５は画像取込センサ３３のプリチャージ及び撮像時のタイミング図、図１６は画像取込センサ３３の撮像データ出力時のタイミング図である。
【００４３】
通常の表示を行う場合には、図１３のモードｍ１の動作を行う。一方、画像取込センサ３３による画像取込を行う場合は、まずモードｍ１の動作を行い、全画素の輝度を所定値（液晶透過率が最も高くなるようにする）に設定する。この場合、図１４に示すように、まず、走査線G1,G4,G7,…を駆動して画面の1/3の表示を行った後、走査線G2,G5,G8,…を駆動して画面の残り1/3の表示を行い、最後に、走査線G3,G6,G9,…を駆動して画面の最後の1/3の表示を行う。そしてバックライトを特定の色で点灯する。本実施形態ではまず白色を点灯する。
【００４４】
次に、モードｍ２で、全画素のキャパシタＣ１をプリチャージ（初期電荷の蓄積）した後、撮像を行う。このとき、図１５に示すように、走査線駆動回路２３が全走査線を駆動している間に、全画素のキャパシタＣ１に５Ｖを書き込む。
【００４５】
次に、モードｍ３で、一部の撮像データ（全画面の12分の1）の出力を行う。具体的には、走査線駆動回路２３のシフトパルスに基づいて所定の走査線をオンすることにより、当該行に属するSRAM３４に保持されたデータが信号線に書き込まれる。この場合、図１６に示すように、まず、走査線G1,G4,G7,…に接続された画素内の画像取込センサ３３の撮像データが信号線に出力される。残りの撮像データ（全画面の12分の11）すなわち、走査線G1,G4,G7,…に接続された画素内の画像取込センサ３３の撮像データがのうちまだラッチ９７に保持されているだけで出力されずにいるデータの出力、走査線G2,G5,G8,…に接続された画素内の画像取込センサ３３の撮像データの信号線への出力、及び走査線G3,G6,G9,…に接続された画素内の画像取込センサ３３の撮像データの信号線への出力はモードｍ４で行う（モードｍ３ではこれらは行わない）。
【００４６】
信号線上に出力された撮像データは、図１５のP/S変換回路９１内のラッチ回路９７に保持される。HSW[3:0]を(1,0,0,0)とすることにより、４つのラッチ回路９７のうちいずれか一つのデータがシフトレジスタに書き込まれる。シフトレジスタ列をクロック(HCK)駆動することにより順に出力される。
【００４７】
まず最初は、1,4,…,238行のデータのうち、1,5,9,…列のデータの出力が出力される。これは、全画素データの1/12に相当する。ここまでのデータに基づいて平均階調Ｌmeanを計算する。この動作の際には、LCDC２側では平均階調Lmeanをカウントする。
【００４８】
全画素データの1/12の平均階調が飽和していないか否かを判定し（ステップＳ１１）、飽和している場合は、データ出力を中止して、画像処理に移行する（モードｍ５）。
【００４９】
次に、平均階調が小さすぎないか否かを判定し（ステップＳ１２）、小さすぎる場合には、次の撮像時間をT+2×ΔTと長めにしてモードｍ２以降の処理を繰返す。小さすぎない場合には、平均階調が大きすぎないか否かを判定し（ステップＳ１３）、大きすぎる場合には、次の撮像時間をT＋0.5×ΔTと短めにしてモードｍ２以降の処理を繰り返す。大きすぎない場合には、モードm4により、残り１２分の１１のデータ出力を継続して行う。
【００５０】
以上のモードm1からモードm4の動作を、平均階調が飽和してしまうまで繰り返す。
【００５１】
モードm5では、こうして得られた撮像データを平均化処理することにより、白色成分の階調情報を合成することができる。
【００５２】
同様にｍ４〜ｍ７で緑色成分の合成と、青色成分の合成とを行う。白色、緑、青は、バックライト（LED）の発光色を白にするか、緑にするか、青にするかで切り替える。
【００５３】
ここでは、バックライトを赤色点灯した状態では撮像は略することができる。合成された白色成分から、合成された青成分及び緑成分を減算することにより、赤成分を合成できる。画像取込センサ３３の光電流は波長分散を有し、赤色の光を検出するには撮像時間を長くする必要がある場合に、全体の撮像時間が長くなってしまう問題を防ぐことができる。
【００５４】
上述した手法により、RGBの各色の階調情報が求まった場合には、これら各色の合成結果を重ね合わせることにより、カラー撮像画面を合成できる。このカラー撮像画面はLCDC２の画像メモリ上に格納され、ベースバンドLSI３を経由して画像処理IC５に送られる。そして、汎用的な画像処理（階調補正、色補正、欠陥画素補正、エッジ補正、ノイズ除去、ホワイトバランス補正など）が行われ、再度LCDC２の表示用のフレームメモリ１６に所定の手順で格納され、LCDC２からLCDに所定のフォーマットで出力することにより、LCDに表示することができる。
【００５５】
図１７はLCDC２の処理動作を示すフローチャートである。図１３で説明した表示装置全体の動作のうち、撮像の際にLCDC2が具体的に行う処理動作を抜き出したものである。LCDC２は、撮像時間T=T+ΔTで撮像するよう画像取込センサ３３に対して指示する（ステップＳ２１）。次に、画像取込センサ３３の撮像データのうち、水平方向は信号線のｍ本ごとに、垂直方向は走査線のｎ本ごとに、画像取込センサ３３の撮像データを取り込む（ステップＳ２２）。これにより、全画素のＭ（＝ｍ×ｎ）分の１個の撮像データを取り込み、撮像データの平均階調Lmeanを計算する。（上述の実施形態においては、m=4、n=3の例を説明したが、m,nはこれらに限定されない）
【００５６】
次に、平均階調Lmeanが所定の基準値（例えば、"64"）以下か否かを判定する（ステップＳ２３）。基準値以下の場合には、直前の撮像データの平均階調Lmean0との差異が所定の基準値ΔH0以上か否かを判定する（ステップＳ２４）。
【００５７】
差異が基準値以上であれば、差異が所定の基準値ΔH1以下か否かを判定する（ステップＳ２５）。差異が基準値ΔH1以下であれば、残りの画像取込センサ３３の撮像データを順に取り込み、画像処理メモリ６５に格納されている各画素の撮像データに加算する（ステップＳ２６）。次に、通算の撮像回数Ａを"1"カウントアップした（ステップＳ２７）後、ステップＳ２１以降の処理を繰り返す。
【００５８】
一方、ステップＳ２４で差異が基準値ΔH0未満と判定された場合、またはステップＳ２５で差異が基準値ΔH1より大きいと判定された場合には、ステップＳ２１に戻る。
【００５９】
また、ステップＳ２３で平均階調Lmeanが64より大きいと判定された場合は、座標(x,y)の画素の階調値L(x,y)を（２）式に基づいて求める。
L(x,y)＝L(x,y)／Ａ …（２）
【００６０】
図１８は図９のステップＳ３の処理の詳細フローチャートである。まず、基準パターンである白色板を、液晶を中間調に設定した状態で画像取込センサ４５で撮像する（ステップＳ３１）。白色板は白紙でもよいし、硫酸バリウム等を塗布した平らな板でもよい。
【００６１】
より具体的には、白色板を撮像時間T1（例えば５msec）〜T2（例えば50msec）の範囲で、等時間間隔で撮像時間を変えながら64回撮像し、撮像結果を画素ごとに単純平均して、多階調データを生成する。撮像時間が最短のT1では、ほとんどの撮像対象が真っ黒（L0とする）と認識され、撮像時間が最長のT2では、ほとんどの撮像対象が真っ白と認識されるように、各撮像時間を設定する。
【００６２】
このような手順で撮像した白色板の再現撮像画像は図１９のようになる。図１９は中間調背景に黒っぽい部分と白っぽい部分とが入り乱れて「ムラ」が生じている再現画像である。図１９からわかるように、ムラのない白色板を撮像したにもかかわらず、撮像データにはムラが生じている。ムラが生じる理由は、各画素のセンサのリーク電流にばらつきがあるためと、各画素のSRAMの動作しきい値にばらつきがあるためである。
【００６３】
このため、比較的長い撮像時間でも白と認識しない画素や、その逆に比較的短い撮像時間でも白と認識する画素などが存在する。前者の画素では再現撮像画像が黒っぽくなり、後者の画素では再現撮像画像が白っぽくなる。
【００６４】
本実施形態では、白色板を撮像した多階調データを図７の画像処理用メモリ４５に格納する(ステップＳ３２）。ここで、メモリ容量を削減するために、ムラの差分値だけを格納してもよい。全画面でムラは±４階調分程度しかないため、ムラの差分値は３ビット程度で表現できる。したがって、階調値をそのまま格納する場合に比べて、画像処理用メモリ４５のメモリ容量を節約できる。
【００６５】
なお、基準パターンの色は必ずしも白色でなくてもよく、中間調や、中間調よりも黒っぽい色でもよい。
【００６６】
続いて、実際の撮像対象物を図１の表示装置で撮像する（ステップＳ３３）。この場合も、撮像条件を変えて複数回撮像を行った結果を平均化して、多階調データを生成する。生成した多階調データは画像処理用メモリ４５に格納される（ステップＳ３４）。そのまま再現したのが図２０である。図２０は現実の撮像対象に図１９のムラを重ね合わせたような、ムラのある再現画像である。
【００６７】
続いて、ステップＳ３４で格納した多階調データから、ステップＳ３２で得られた多階調データを減じてムラ成分を除去する（ステップＳ３５）。このとき、ステップＳ３２で格納した白色板の多階調データをそのまま利用するのではなく、階調値に応じて重み付けを行ってもよい。
【００６８】
より具体的には、例えば（３）式に基づいてムラ成分を除去する。
Ｃ(x,y)＝Ｌ(x,y)−｛Ｂ(x,y)−average(Ｂ(x,y))｝×ｆ(Ｌ(x,y)) …（３）
【００６９】
ここで、(x, y)は画素位置、Ｂ(x, y)は基準パターンの多階調データ、average（B(x,y)）はB(x,y)を画面全体で平均した値（平均階調）、Ｌ(x, y)は撮像対象物のムラ除去前の多階調データ、Ｃ(x, y)は撮像対象物のムラ除去後の階調値である。
【００７０】
ムラ成分が除去された多階調データは画像処理用メモリ４５に格納されるとともに、適当なタイミングで画素アレイ部２１に表示される（ステップＳ３６）。
【００７１】
図２１は白色板を撮像して得られた多階調データの階調値の変化を示す図であり、Ｘ軸は図１９の水平方向座標、Ｙ軸は階調値を表している。図示のように、白色板であるにもかかわらず、階調値は点線で示す階調値を挟んで上下に変動する。
【００７２】
図２２は図２０に示す実際の撮像対象物を撮像して得られる多階調データの階調値の変化を示す図であり、Ｘ軸は図２０の水平方向座標、Ｙ軸は階調値を表している。図２２の実線曲線ｃｂ１が撮像対象物の多階調データ、点線ｃｂ２は白色板の多階調データである。
【００７３】
上述した（３）式では、白色板の多階調データを画面全体で平均化した画素値average(Ｂ(x, y))を計算し、白色板の多階調データＢ(x, y)と平均階調値average(Ｂ(x, y))との差分を、画素ごとに重み付けする。そして、この重み付けされた差分を、撮像対象物の多階調データから減じることにより、ムラの除去を行う。
【００７４】
この結果、図２３に示すようにムラのない画像が得られ、階調値の変動も図２４に示すようにほとんどなくなる。図２３は現実の撮像対象に近い、ムラのない再現画像である。
【００７５】
次に、重み付け関数について詳述する。図２５は撮像対象物の階調値と撮像対象物を撮像して得られる多階調データの階調誤差との関係を示す図であり、縦軸が階調誤差を表している。ここでは一例として、L０〜L63の全64階調のうち、L5,L32,L60の3階調について模式的に図示した。図２５の例では、撮像対象物を64階調で表し、L63を真っ白とし、L0を真っ黒としている。図２５のL60、L32、L5の順に白っぽい階調になる。図示のように、白に近いほど階調誤差は少なく、黒に近いほど階調誤差は大きくなる。この傾向は、本実施形態の撮像シーケンス（光源輝度は一定として撮像時間のみ段階的に変更して複数の撮像を行う）の場合に限らず、別の撮像シーケンス（例：撮像時間は一定として光源輝度のみ段階的に変更して複数の撮像を行う）の場合にも同様の傾向を示す。撮像対象の暗い部分では、明るい部分に比べ光リークの進行が遅く、光電変換素子の光リーク電流のばらつきの差や画素内回路を構成するTFTの特性ばらつきが大きくなるためである。したがって減算の際の重み付け関数は階調についての単調減少関数とするのがよい。
【００７６】
このため、図２６に示すように、撮像対象物の階調値に応じて、線形に重み付け係数が変化するような一次関数からなる重み付け関数を用いてもよい。
【００７７】
また、現実に撮像を行う場合は、撮像対象物によって、異なる撮像条件で撮像を行う場合がある。例えば、暗めの撮像対象物を撮像する場合は撮像時間を長めに、明るめの撮像対象物を撮像する場合は撮像時間を短めにすることにより再現画像のコントラスト等の画質が好ましくなることがある。このように、撮像条件が異なる場合には、必ずしも図２６のような一次の重み関数で重み付け係数を求めるのが望ましくない場合もあるし、図２６のような重み付け関数を用いることで、新たなムラが発生するおそれもある。
【００７８】
また、液晶自体に図２７のような非線形のガンマ特性があるため、その影響も考慮に入れて重み付け係数を設定するのが望ましい場合もある。
【００７９】
このため、場合によっては、図２８に示すように、重み付け関数を折れ線関数にしてもよい。図２８の例では、中間調付近で、一次の重み付け関数の傾きを変化させる例を示している。傾きを変化させるポイントは必ずしも中間調付近には限定されず、撮像条件や液晶のガンマ特性などにより決めるのが望ましい。
【００８０】
あるいは、図２９に示すように、２次以上の多次関数で重み付け関数を構成してもよい。図２９の重み付け関数は、階調値に応じて重み付け係数が単調変化し、中間調付近の重み付け係数の変化量を小さくしている。
【００８１】
このようにすることによって、図１９のような減算処理のためのムラパターンの撮像を、都度頻回に行うことなく、表示装置の出荷時やユーザーにおけるシステムセットアップ時に1回だけ行うだけでよくなる。
【００８２】
上述したムラの除去処理は、RGBの各色ごとに個別に行われ、その後に撮像画像の表示が行われる。
【００８３】
このように、本実施形態では、センサで撮像して得られた多階調データに含まれるムラ成分を、上述した（１）式に基づいて除去するため、センサの特性ばらつき等により発生するムラの影響を受けない撮像データが得られる。
【００８４】
上述した実施形態では、カラーフィルタを設けないで、バックライトの発光色を赤→青→緑と所定周期で切替えることによりカラー表示を行う表示装置において、図３に示すように各画素ごとに画像取込センサを設ける例を説明したが、カラーフィルタを各画素に設け、１画素を構成する３色（RGB）のうち、１色または２色分だけセンサを配置してもよい。
【００８５】
例えば、図３０は１画素を構成する３色のうち、緑色だけにセンサを設ける例を示している。この場合の１画素分の回路構成は図３１のような回路図で表される。
【００８６】
緑色に対応する画素TFT３１には、初期化用TFT３５を介して画像取込センサ３３が接続されている。ところが、赤色に対応する画素TFTと青色に対応する画素TFTには画素取込センサは接続されていない。
【００８７】
図３１のような構成にすることにより、画像取込センサの数を削減でき、構造を簡易化できるとともに、液晶表示を行う際の開口率の向上が図れる。
【００８８】
【発明の効果】
以上詳細に説明したように、本発明によれば、被写体の撮像結果である第１の多階調データに含まれるムラを、基準パターンの撮像結果である第２の多階調データを用いて調整するため、ムラのない多階調データが得られる。このため、撮像部の特性により撮像結果にムラが生じても、そのムラを相殺できる。
【図面の簡単な説明】
【図１】本発明に係る表示装置の全体構成を示すブロック図。
【図２】 LCD基板１上に形成される回路を示すブロック図。
【図３】画素アレイ部２１の１画素分の詳細回路図。
【図４】ガラス基板上の１画素分のレイアウト図。
【図５】画像取込の方法を説明する図。
【図６】画像処理IC５の内部構成を示すブロック図。
【図７】 LCDC２の内部構成の一例を示すブロック図。
【図８】従来のLCDC２の内部構成を示すブロック図。
【図９】 LCDC２が行う画像取込時の処理手順を示すフローチャート。
【図１０】逐次加算方法を説明する図。
【図１１】 LCD基板１上の信号線駆動回路２２、走査線駆動回路２３、センサ制御回路２４及び信号処理出力回路２５と、LCDC２と、ベースバンドLSI３との間の信号のやり取りを示す図。
【図１２】ガラス基板の詳細構成を示すブロック図。
【図１３】本実施形態の表示装置の動作を説明する図。
【図１４】通常表示時のタイミング図。
【図１５】センサ３３のプリチャージ及び撮像時のタイミング図。
【図１６】センサ３３の撮像データ出力時のタイミング図。
【図１７】 LCDC２の処理動作を示すフローチャート。
【図１８】図９のステップＳ３の処理の詳細フローチャート。
【図１９】白色板の再現撮像画像を示す図。
【図２０】実際の撮像対象物の撮像画像を示す図。
【図２１】白色板を撮像して得られた多階調データの階調値の変化を示す図。
【図２２】図２０に示す撮像画像の階調値の変化を示す図。
【図２３】図２０からムラを除去した画像を示す図。
【図２４】図２３に対応する階調値の変化を示す図。
【図２５】撮像対象物の階調値と撮像対象物を撮像して得られる多階調データの階調誤差との関係を示す図。
【図２６】一次関数からなる重み付け関数を示す図。
【図２７】非線形のガンマ特性を示す図。
【図２８】折れ線からなる重み付け関数を示す図。
【図２９】多次関数からなる重み付け関数を示す図。
【図３０】１画素を構成する３色のうち、緑色だけにセンサを設ける例を示す図。
【図３１】１画素の回路構成を示す回路図。
【符号の説明】
１ LCD基板
２ LCDC
３ ベースバンドLSI
４ カメラ
５ 画像処理IC
６ 送受信部
７ 電源回路
１１ CPU
１２ メインメモリ
１３ MPEG処理部
１４ DRAM
１５ 制御部
１６ フレームメモリ
２１ 画素アレイ部
２２ 信号線駆動回路
２３ 走査線駆動回路
２４ センサ制御回路
２５ 信号処理出力回路
３１ 画素TFT
３２ 表示制御TFT
３３ 画像取込センサ
３４ SRAM
３５ 初期化用TFT[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a display device having an image capturing function.
[0002]
[Prior art]
The liquid crystal display device includes an array substrate on which signal lines, scanning lines, and pixel TFTs are arranged, and a drive circuit that drives the signal lines and the scanning lines. With the recent progress and development of integrated circuit technology, a process technology for forming a part of a drive circuit on an array substrate has been put into practical use.
[0003]
As one of such process technologies, a low-temperature polysilicon TFT (Thin Film Transistor) process technology has attracted attention. By adopting a low-temperature polysilicon TFT process, the entire liquid crystal display device can be made light and thin, and it is widely used as a display device for various portable devices such as mobile phones and notebook computers.
[0004]
By the way, a display device having an image capturing function in which a contact area sensor for capturing an image is arranged on an array substrate has been proposed (see, for example, Patent Documents 1 and 2).
[0005]
In a conventional display device having this type of image capturing function, an image capturing sensor is arranged for each pixel, charges corresponding to the amount of light received by the image capturing sensor are accumulated in the capacitor, and the voltage across the capacitor is stored. It detects and captures an image.
[0006]
[Patent Document 1]
Japanese Patent Laid-Open No. 2001-292276
[Patent Document 2]
JP 2001-339640 A
[0007]
[Problems to be solved by the invention]
However, when the above-described sensor is formed by the low-temperature polysilicon TFT process, even if a single color subject is imaged due to individual variations of the sensor, the gradation value of the imaged data varies depending on the location, and the imaged data is displayed. Display irregularities occur.
[0008]
As a countermeasure for suppressing such display unevenness, Patent Document 2 described above discloses a technique of repeatedly performing imaging and setting an optimal imaging time for each horizontal line.
[0009]
Since this technique is complicated to control, it may take time to capture an image. In addition, a high-performance CPU and controller are required, which increases costs.
[0010]
The present invention has been made in view of such a point, and an object thereof is to provide a display device capable of suppressing unevenness of imaging data by simple processing.
[0011]
[Means for Solving the Problems]
In order to solve the above-described problem, in one embodiment of the present invention, a display element in a pixel formed in the vicinity of each intersection of a signal line and a scanning line arranged in rows and columns,
An imaging unit provided at least one for each of the display elements, each for imaging a predetermined range of a subject;
An initial multi-grayscale data storage unit for storing first multi-grayscale data corresponding to the imaging result of the imaging unit;
A reference pattern storage unit that stores second multi-gradation data obtained by imaging a predetermined reference pattern by the imaging unit;
An unevenness adjustment data generating unit that generates third multi-gradation data in which display unevenness of the first multi-gradation data is adjusted based on the first and second multi-gradation data,
The unevenness adjustment data generation unit
For each pixel, a difference calculation unit that calculates a difference between the second multi-gradation data and the gradation average value of the second multi-gradation data for one screen;
A weight calculation unit that calculates a weighting coefficient corresponding to the first multi-gradation data for unevenness adjustment set for each pixel for each pixel;
For each pixel, a non-uniformity adjustment value calculation unit that calculates the non-uniformity adjustment value by multiplying the corresponding weighting coefficient by the difference,
And a non-uniformity canceling unit that generates the third multi-gradation data by subtracting the non-uniformity adjustment value from the first multi-gradation data for each pixel.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, a display device according to the present invention will be specifically described with reference to the drawings.
[0013]
FIG. 1 is a block diagram showing the overall configuration of a display device according to the present invention, and shows the configuration of a display device of a camera-equipped mobile phone. 1 includes an LCD (Liquid Crystal Display) substrate 1 in which pixel TFTs are arranged, a liquid crystal driver IC (hereinafter referred to as LCDC) 2 mounted on the LCD substrate 1, a baseband LSI 3, and a camera 4. And an image processing IC 5 that performs image processing of image data captured by the camera 4, a transmission / reception unit 6 that communicates with the base station, and a power supply circuit 7 that supplies power to each unit.
[0014]
The baseband LSI 3 includes a CPU 11, a main memory 12, an MPEG processing unit 13, a DRAM 14, an audio signal processing unit (not shown), and the like, and controls the entire mobile phone. In FIG. 1, the image processing IC 5 and the transmission / reception unit 6 are provided separately from the baseband LSI 3, but these may be combined into one chip. The CPU 11 and the main memory 12 may be one chip, and the other may be another chip.
[0015]
The LCDC 2 includes a control unit 15 and a frame memory 16. The camera 4 is realized by a CCD (Charge Coupled Device) or a CMOS image capture sensor.
[0016]
The LCD substrate 1 of the present embodiment is provided with an image capture sensor for capturing an image for each pixel. On the LCD substrate 1, a common substrate is formed of a transparent electrode such as ITO, and a counter substrate is arranged at a predetermined interval (about 5 microns), a liquid crystal material is injected between them and sealed by a predetermined method. A polarizing plate is attached to the outside of the substrate.
[0017]
FIG. 2 is a block diagram showing a circuit formed on the LCD substrate 1. As shown in the figure, on the LCD substrate 1, a pixel array section 21 in which signal lines and scanning lines are arranged, a signal line driving circuit 22 for driving signal lines, and a scanning line driving circuit 23 for driving scanning lines. Then, an image capturing sensor control circuit 24 that controls image capturing and a signal processing output circuit 25 that performs signal processing after image capturing are formed. These circuits are formed by, for example, a polysilicon TFT using low-temperature polysilicon technology. The signal line driving circuit 22 includes a D / A conversion circuit that converts digital pixel data into an analog voltage suitable for driving the display element. A known D / A conversion circuit is used.
[0018]
FIG. 3 is a detailed circuit diagram for one pixel of the pixel array section 21, and FIG. 4 is a layout diagram for one pixel on the glass substrate. As shown in FIG. 4, the pixel of this embodiment is substantially square.
[0019]
As shown in FIG. 3, each pixel has a pixel TFT 31, a display control TFT 32 that controls whether or not electric charge is accumulated in the auxiliary capacitor Cs, an image capturing sensor 33, and an image capturing result of the image capturing sensor 33. The capacitor C1 to be stored, the SRAM 34 for storing binary data corresponding to the accumulated charge of the capacitor C1, and the initialization TFT 35 for accumulating the initial charge in the capacitor C1.
[0020]
Here, the luminance of each pixel is determined by the difference between the pixel electrode potential determined based on the electric charge accumulated in the auxiliary capacitor Cs and the potential of the common electrode formed on the counter substrate. The gradation is controlled by controlling the transmittance.
[0021]
Although FIG. 3 shows an example in which one image capturing sensor 33 is provided for each pixel, the number of image capturing sensors 33 is not particularly limited. As the number of image capturing sensors 33 per pixel is increased, the resolution of image capturing can be improved.
[0022]
When the capacitor C1 is initialized, the pixel TFT 31 and the initialization TFT 35 are turned on. When an analog voltage (analog pixel voltage) for setting the luminance of the display element is written to the auxiliary capacitor Cs, the pixel TFT 31 and the display control TFT 32 are turned on. When refreshing the voltage of the capacitor C1, both the initialization TFT 35 and the data holding TFT 36 in the SRAM 34 are turned on. If the voltage of the capacitor C1 is close to the power supply voltage (5V) of the SRAM 34, the result of the refresh will be 5V even if there is some leakage, and conversely, the voltage of the capacitor C1 will be close to the GND voltage (0V) of the SRAM 34. If there is, it becomes 0V as a result of the refresh. Further, as long as both the TFT 35 and the TFT 36 are on, the data value of the SRAM 34 continues to be held extremely stably. Even if either TFT 35 or TFT 36 is turned off, the data value of the SRAM 34 continues to be held as long as the potential leakage of the capacitor C1 is small. If refresh is performed before the potential leak of the capacitor C1 increases and the data value does not change, the data value of the SRAM 34 can be held. When supplying imaging data stored in the SRAM 34 to the signal line, both the pixel TFT 31 and the data holding TFT 36 are turned on.
[0023]
The display device of the present embodiment can perform a normal display operation, and can also capture an image similar to a scanner. When a normal display operation is performed, the TFTs 35 and 36 are set to an off state, and valid data is not stored in the buffer. In this case, the signal line voltage from the signal line driving circuit 22 is supplied to the signal line, and display according to the signal line voltage is performed.
[0024]
On the other hand, when capturing an image, as shown in FIG. 5, an image capturing object (for example, a paper surface) 37 is disposed on the upper surface side of the LCD substrate 1, and light from the backlight 38 is transmitted to the counter substrate 39 and the LCD substrate 1. Irradiates the paper surface 37 via The light reflected by the paper surface 37 is received by the image capture sensor 33 on the LCD substrate 1 and image capture is performed. Here, the glass substrate and the polarizing plate arranged on the imaging target side should be as thin as possible. Desirably, the total is about 0.2 mm or less. The paper surface is usually a diffuse reflection surface, and diffuses the irradiated light. If the glass substrate on the imaging target side is thick, the distance between the light receiving unit of the image capturing sensor and the paper surface will increase, and diffused reflected light will easily enter the image capturing sensor of the adjacent pixel, which may cause the captured image to blur. It is.
[0025]
The captured image data is stored in the SRAM 34 as shown in FIG. 3, and then sent to the LCDC 2 shown in FIG. 1 through a signal line. The LCDC 2 receives the digital signal output from the display device according to the present embodiment, and performs arithmetic processing such as data rearrangement, removal of unevenness in the captured image, and removal of noise in the data.
[0026]
FIG. 6 is a block diagram showing the internal configuration of the image processing IC 5. The image processing IC 5 in FIG. 6 receives the imaging data from the camera I / F unit 41 that receives the imaging data captured by the camera 4, the control unit 42, the control I / F 43 that controls the operation of the camera 4, and the LCDC 2. Receiving LCD-I / F 44, image processing memory 45 for storing imaging data, host I / F 46 for exchanging control signals with the CPU 11, and gradation correction of imaging data (for removing unevenness) Gradation correction unit 47 that performs processing), color correction unit 48 that performs color correction of imaging data, defective pixel correction unit 49, edge correction unit 50 that performs edge correction of imaging data, and noise in imaging data are removed. And a white balance correction unit 52 that adjusts the white balance of the imaging data. A difference from the conventional image processing IC is that it has an LCD-I / F 44 that receives imaging data output from the LCD substrate 1.
[0027]
In principle, the display on the LCD substrate 1 is performed under instructions and monitoring from the baseband LSI 3. For example, when imaging data of the camera 4 is input to the baseband LSI 3, the baseband LSI 3 outputs the imaging data to the LCDC 2 at a predetermined timing. The LCDC 2 stores the image data of the camera 4 from the baseband LSI 3 in the frame memory 16. Even if the imaging data of the camera 4 supplied from the baseband LSI 3 is intermittent, the LCDC 2 outputs the imaging data of the camera 4 for one screen stored in the frame memory 16 to the LCD substrate 1 at a predetermined timing. To do. The LCD substrate 1 converts the imaging data of the camera 4 from the LCDC 2 into an analog pixel voltage and writes it to the signal line.
[0028]
FIG. 7 is a block diagram showing an example of the internal configuration of the LCDC 2. The LCDC 2 in FIG. 7 includes an MPEG-I / F 61, a LUT (Lookup Table) 62, an LCD-I / F 63, a line buffer 64 for storing imaging data, and image processing for holding imaging data supplied from the LCDC 2. Memory 65, frame memory 16 holding digital pixel data for display, pre-output operation unit 66, first buffer 67, second buffer 68, image processing unit 69, host I / F 70, oscillator 71.
[0029]
On the other hand, FIG. 8 is a block diagram showing an internal configuration of the conventional LCDC 2. As shown, the conventional LCDC 2 includes an MPEG-I / F 61, an LUT 62, an LCD-I / F 63, a frame memory 16, a buffer 67, and an oscillator 71.
[0030]
Conventionally, when displaying a moving image, an MPEG codec signal input via the MPEG-I / F is converted into RGB data with reference to the LUT 62 and stored in the frame memory 16. When displaying text, the drawing command supplied from the CPU 11 via the host I / F 45 is converted into RGB data and stored in the frame memory 16. The oscillator 71 generates a reference clock as necessary. When the standby screen must be displayed when the CPU is idle, such as when waiting for a mobile phone, the pixel data for display is constantly sent from the LCDC 2 to the LCD substrate 1 in synchronization with the reference clock. Continue.
[0031]
The LCDC 2 rearranges the digital image data read from the frame memory 16 as necessary, for example, one line at a time from the first line of the display screen, and outputs it to the LCD substrate 1.
[0032]
As shown in FIG. 7, the LCDC 2 of the present embodiment includes an image processing memory 65 that the conventional LCDC 2 does not have, and an image capture sensor 33 supplied from the LCD substrate 1 via the LCD-I / F 43. Image data is stored. The imaging data of the image capture sensor 33 is supplied to the image processing IC 5 via the host I / F 45 and the baseband LSI 3.
[0033]
Since each pixel in the LCD substrate 1 must secure an aperture ratio, a space for arranging an image capturing sensor 33 for image capturing and peripheral circuits is limited. If the aperture ratio is small, the backlight must be lit at a higher brightness in order to ensure the display brightness of the screen during normal display, resulting in a problem that the power consumption of the backlight increases. is there. It is desirable to keep as few image capture sensors 33 and associated circuits as possible in each pixel. Further, even if there is only one image capturing sensor 33, if a subtle change in the potential of the capacitor C1 caused by the image capturing sensor 33 can be accurately extracted to the outside, multi-gradation image capturing can be realized thereby, but it is difficult. It is. Because TFTs and image capture sensors 33 formed on a glass substrate have non-negligible variations in operation thresholds even on the same substrate, an analog amplifier circuit with a small error for outputting an analog signal as it is from a pixel is provided. Because it is difficult. Further, although it is conceivable to provide a variation compensation circuit in the pixel, there is a problem that the variation compensation circuit itself occupies a certain area and impairs the aperture ratio. Therefore, in order to capture a multi-tone image, a plurality of image capturing sensors 33 are not provided in the pixel or a complicated compensation circuit is not provided, and a plurality of imaging operations are performed while changing the imaging conditions. Based on these data, processing for multi-gradation and processing for noise compensation were performed.
[0034]
FIG. 9 is a flowchart showing a processing procedure at the time of image capture performed by the LCDC 2. First, image capture by the image capture sensor 33 is performed N times while changing the imaging conditions (step S1). Next, a simple average of N times of imaging data is calculated based on the equation (1) (step S2). Here, L (x, y) i represents the gradation value of the i-th coordinate (x, y).
[Expression 1]

[0035]
When performing the processing of steps S1 and S2, as shown in FIG. 10, it is necessary to perform sequential addition for sequentially adding the gradation values of each time, and after performing sequential addition up to the Nth time, divide by N. It is not necessary to hold the imaging data that has already been added in the process of sequential addition.
[0036]
When performing the sequential addition as shown in FIG. 10, the frame memory 16 only needs to have a capacity capable of storing about two times of image data, and the memory capacity can be reduced.
[0037]
Next, unevenness pattern subtraction processing is performed (step S3). Next, white balance adjustment, defect correction, and the like are performed (step S4).
[0038]
11 shows signal exchange between the signal line driving circuit 22, the scanning line driving circuit 23, the image capturing sensor control circuit 24 and the signal processing output circuit 25 on the LCD substrate 1, and the LCDC 2 and the baseband LSI 3. FIG.
[0039]
FIG. 12 is a block diagram showing a detailed configuration of the glass substrate. The pixel array unit 21 of the present embodiment has a display resolution of 320 pixels in the horizontal direction × 240 pixels in the vertical direction. A so-called field sequential drive is performed in which the backlight emits light sequentially in red, green and blue. In the field sequential driving, the light emission color of the backlight may be lit white as well as red, green and blue. Each pixel is provided with a signal line, a scanning line, and the like. The total number of signal lines is 320, and the total number of scanning lines is 240.
[0040]
The scanning line driving circuit 23 includes a 240-stage shift register 71, a three-select decoder 72, a level shifter (L / S) 73, a multiplexer (MUX) 74, and a buffer 75.
[0041]
The signal processing output circuit 25 includes 320 precharge circuits 76, a 4-select decoder 77, a total of 80 shift registers 78 to which a data bus is connected every 10 stages, and eight output buffers 79. Have.
[0042]
FIG. 13 is a diagram for explaining the operation of the display device of the present embodiment, FIG. 14 is a timing diagram during normal display, FIG. 15 is a timing diagram during precharge and imaging of the image capture sensor 33, and FIG. It is a timing chart at the time of imaging data output of the sensor 33.
[0043]
When performing normal display, the operation in the mode m1 in FIG. 13 is performed. On the other hand, when the image capturing by the image capturing sensor 33 is performed, first, the operation of the mode m1 is performed, and the luminance of all the pixels is set to a predetermined value (the liquid crystal transmittance is maximized). In this case, as shown in FIG. 14, first, the scanning lines G1, G4, G7,... Are driven to display 1/3 of the screen, and then the scanning lines G2, G5, G8,. The remaining 1/3 of the screen is displayed. Finally, the scanning lines G3, G6, G9,... Are driven to display the last 1/3 of the screen. Then, the backlight is turned on with a specific color. In this embodiment, first, white is lit.
[0044]
Next, in mode m2, imaging is performed after the capacitors C1 of all the pixels are precharged (initial charge accumulation). At this time, as shown in FIG. 15, while the scanning line driving circuit 23 drives all the scanning lines, 5 V is written to the capacitors C1 of all the pixels.
[0045]
Next, in mode m3, a part of the imaging data (1/12 of the entire screen) is output. Specifically, by turning on a predetermined scanning line based on the shift pulse of the scanning line driving circuit 23, the data held in the SRAM 34 belonging to the row is written to the signal line. In this case, as shown in FIG. 16, first, the image data of the image capturing sensor 33 in the pixels connected to the scanning lines G1, G4, G7,... The remaining imaging data (11/12 of the entire screen), that is, the imaging data of the image capturing sensor 33 in the pixels connected to the scanning lines G1, G4, G7,. The output of the data that is not output only, the output of the imaging data of the image capturing sensor 33 in the pixels connected to the scanning lines G2, G5, G8,. ,... Are output in the signal line of the imaging data of the image capturing sensor 33 in the pixels connected to.
[0046]
The imaging data output on the signal line is held in the latch circuit 97 in the P / S conversion circuit 91 of FIG. By setting HSW [3: 0] to (1,0,0,0), any one of the four latch circuits 97 is written into the shift register. The shift register trains are sequentially output by driving the clock (HCK).
[0047]
First, the output of the data of 1,5,9,... Column among the data of 1,4,. This corresponds to 1/12 of the total pixel data. The average gradation Lmean is calculated based on the data so far. In this operation, the average gradation Lmean is counted on the LCDC 2 side.
[0048]
It is determined whether or not 1/12 average gradation of all pixel data is saturated (step S11). If saturated, data output is stopped and the process proceeds to image processing (mode m5). .
[0049]
Next, it is determined whether or not the average gradation is too small (step S12). If the average gradation is too small, the next imaging time is lengthened to T + 2 × ΔT and the processing after mode m2 is repeated. If it is not too small, it is determined whether or not the average gradation is too large (step S13). If it is too large, the next imaging time is shortened to T + 0.5 × ΔT and the processing after mode m2 is performed. repeat. If it is not too large, the remaining 11th of the data is continuously output in mode m4.
[0050]
The above operations from mode m1 to mode m4 are repeated until the average gradation is saturated.
[0051]
In mode m5, the gradation information of the white component can be synthesized by averaging the imaging data thus obtained.
[0052]
Similarly, the synthesis of the green component and the synthesis of the blue component are performed at m4 to m7. White, green, and blue are switched according to whether the backlight (LED) emission color is white, green, or blue.
[0053]
Here, imaging can be omitted when the backlight is lit in red. The red component can be synthesized by subtracting the synthesized blue component and green component from the synthesized white component. The photocurrent of the image capture sensor 33 has chromatic dispersion, and it is possible to prevent a problem that the entire imaging time becomes long when it is necessary to lengthen the imaging time to detect red light.
[0054]
When the gradation information of each color of RGB is obtained by the above-described method, the color imaging screen can be synthesized by superimposing the synthesis results of these colors. This color imaging screen is stored on the image memory of the LCDC 2 and sent to the image processing IC 5 via the baseband LSI 3. Then, general-purpose image processing (gradation correction, color correction, defective pixel correction, edge correction, noise removal, white balance correction, etc.) is performed and stored again in the display frame memory 16 of the LCDC 2 in a predetermined procedure. The data can be displayed on the LCD by outputting it from the LCDC2 to the LCD in a predetermined format.
[0055]
FIG. 17 is a flowchart showing the processing operation of the LCDC2. Among the operations of the entire display device described with reference to FIG. 13, processing operations specifically performed by the LCDC 2 at the time of imaging are extracted. The LCDC 2 instructs the image capture sensor 33 to image at the imaging time T = T + ΔT (step S21). Next, among the imaging data of the image capturing sensor 33, the imaging data of the image capturing sensor 33 is captured every m signal lines in the horizontal direction and every n scanning lines in the vertical direction (step S22). . Thereby, one M (= m × n) imaging data of all pixels is taken in, and the average gradation Lmean of the imaging data is calculated. (In the above-described embodiment, an example in which m = 4 and n = 3 has been described, but m and n are not limited to these.)
[0056]
Next, it is determined whether or not the average gradation Lmean is equal to or less than a predetermined reference value (for example, “64”) (step S23). If it is less than or equal to the reference value, it is determined whether or not the difference from the average gradation Lmean0 of the immediately preceding imaged data is greater than or equal to a predetermined reference value ΔH0 (step S24).
[0057]
If the difference is greater than or equal to the reference value, it is determined whether or not the difference is less than or equal to a predetermined reference value ΔH1 (step S25). If the difference is equal to or less than the reference value ΔH1, the remaining image capturing data of the image capturing sensor 33 is sequentially captured and added to the image capturing data of each pixel stored in the image processing memory 65 (step S26). Next, after incrementing the total number of times of imaging A by “1” (step S27), the processing after step S21 is repeated.
[0058]
On the other hand, if it is determined in step S24 that the difference is less than the reference value ΔH0, or if it is determined in step S25 that the difference is greater than the reference value ΔH1, the process returns to step S21.
[0059]
If it is determined in step S23 that the average gradation Lmean is greater than 64, the gradation value L (x, y) of the pixel at the coordinate (x, y) is obtained based on equation (2).
L (x, y) = L (x, y) / A (2)
[0060]
FIG. 18 is a detailed flowchart of the process in step S3 of FIG. First, a white plate as a reference pattern is imaged by the image capture sensor 45 with the liquid crystal set to a halftone (step S31). The white plate may be white paper or a flat plate coated with barium sulfate or the like.
[0061]
More specifically, the white plate is imaged 64 times while changing the imaging time at equal time intervals within the imaging time T1 (for example, 5 msec) to T2 (for example, 50 msec), and the imaging result is simply averaged for each pixel. Multi-tone data is generated. Set each imaging time so that most imaging objects are recognized as black (L0) at T1 with the shortest imaging time, and most imaging objects are recognized as white at T2 with the longest imaging time. .
[0062]
A reproduced captured image of the white plate imaged in such a procedure is as shown in FIG. FIG. 19 is a reproduction image in which a blackish portion and a whitish portion are mixed in a halftone background and “unevenness” is generated. As can be seen from FIG. 19, the imaging data is uneven even though the white plate having no unevenness is imaged. The reason why the unevenness occurs is that there is a variation in the leak current of the sensor of each pixel and a variation in the operation threshold value of the SRAM of each pixel.
[0063]
For this reason, there are pixels that are not recognized as white even in a relatively long imaging time, and conversely, pixels that are recognized as white even in a relatively short imaging time. In the former pixel, the reproduced captured image becomes blackish, and in the latter pixel, the reproduced captured image becomes whitish.
[0064]
In the present embodiment, multi-gradation data obtained by imaging a white plate is stored in the image processing memory 45 of FIG. 7 (step S32). Here, in order to reduce the memory capacity, only the unevenness difference value may be stored. Since the unevenness is only about ± 4 gradations on the entire screen, the unevenness difference value can be expressed by about 3 bits. Therefore, the memory capacity of the image processing memory 45 can be saved as compared with the case where the gradation value is stored as it is.
[0065]
Note that the color of the reference pattern does not necessarily have to be white, and may be a halftone or a darker color than the halftone.
[0066]
Subsequently, an actual imaging object is imaged by the display device of FIG. 1 (step S33). Also in this case, multi-gradation data is generated by averaging the results of imaging performed a plurality of times under different imaging conditions. The generated multi-gradation data is stored in the image processing memory 45 (step S34). FIG. 20 shows the reproduction as it is. FIG. 20 is a reproduced image with unevenness such that the unevenness of FIG. 19 is superimposed on an actual imaging target.
[0067]
Subsequently, the multi-gradation data stored in step S34 is subtracted from the multi-gradation data stored in step S34 to remove unevenness components (step S35). At this time, instead of using the multi-tone data of the white plate stored in step S32 as it is, weighting may be performed according to the tone value.
[0068]
More specifically, for example, the unevenness component is removed based on the equation (3).
C (x, y) = L (x, y) − {B (x, y) −average (B (x, y))} × f (L (x, y)) (3)
[0069]
Here, (x, y) is the pixel position, B (x, y) is the multi-tone data of the reference pattern, and average (B (x, y)) is the average value of B (x, y) over the entire screen. (Average gradation), L (x, y) is multi-tone data before unevenness removal of the imaging object, and C (x, y) is a gradation value after unevenness removal of the imaging object.
[0070]
The multi-gradation data from which the uneven component has been removed is stored in the image processing memory 45 and displayed on the pixel array unit 21 at an appropriate timing (step S36).
[0071]
FIG. 21 is a diagram showing changes in gradation values of multi-gradation data obtained by imaging a white plate, where the X axis represents the horizontal coordinate in FIG. 19 and the Y axis represents the gradation value. As shown in the figure, the gradation value fluctuates up and down across the gradation value indicated by the dotted line, regardless of the white plate.
[0072]
FIG. 22 is a diagram showing changes in gradation values of multi-gradation data obtained by imaging the actual imaging object shown in FIG. 20, where the X axis is the horizontal coordinate in FIG. 20 and the Y axis is the gradation value. Represents. The solid line curve cb1 in FIG. 22 is the multi-tone data of the imaging object, and the dotted line cb2 is the multi-tone data of the white plate.
[0073]
In the above equation (3), the pixel value average (B (x, y)) obtained by averaging the multi-tone data of the white plate over the entire screen is calculated, and the multi-tone data B (x, y) of the white plate is calculated. And the average gradation value average (B (x, y)) are weighted for each pixel. Then, unevenness is removed by subtracting the weighted difference from the multi-gradation data of the imaging object.
[0074]
As a result, an image having no unevenness is obtained as shown in FIG. 23, and the fluctuation of the gradation value is almost eliminated as shown in FIG. FIG. 23 is a reproduced image with no unevenness that is close to the actual imaging target.
[0075]
Next, the weighting function will be described in detail. FIG. 25 is a diagram showing the relationship between the gradation value of the imaging object and the gradation error of the multi-gradation data obtained by imaging the imaging object, and the vertical axis represents the gradation error. Here, as an example, among the 64 gray levels of L0 to L63, 3 gray levels of L5, L32, and L60 are schematically illustrated. In the example of FIG. 25, the imaging target is represented by 64 gradations, L63 is pure white, and L0 is black. The tone becomes whitish in the order of L60, L32, and L5 in FIG. As shown in the figure, the closer to white, the smaller the gradation error, and the closer to black, the larger the gradation error. This tendency is not limited to the case of the imaging sequence of the present embodiment (a plurality of imaging is performed by changing only the imaging time stepwise with the light source luminance being constant), but other imaging sequences (eg, the imaging light source is constant and the light source The same tendency is also shown in the case of performing a plurality of images by changing only the luminance stepwise. This is because in the dark part to be imaged, the light leakage progresses slower than in the bright part, and the difference in the light leakage current of the photoelectric conversion element and the characteristic variation of the TFT constituting the circuit in the pixel become large. Therefore, the weighting function at the time of subtraction is preferably a monotone decreasing function for gradation.
[0076]
For this reason, as shown in FIG. 26, a weighting function including a linear function that linearly changes the weighting coefficient in accordance with the gradation value of the imaging target may be used.
[0077]
Moreover, when imaging is actually performed, imaging may be performed under different imaging conditions depending on the imaging object. For example, the image quality such as the contrast of the reproduced image may be preferable by taking a longer imaging time when imaging a dark imaging object and shortening the imaging time when imaging a bright imaging object. As described above, when the imaging conditions are different, it may not always be desirable to obtain the weighting coefficient with a primary weighting function as shown in FIG. 26, or a new weighting function as shown in FIG. There is also a risk of unevenness.
[0078]
Further, since the liquid crystal itself has a non-linear gamma characteristic as shown in FIG. 27, it may be desirable to set the weighting factor in consideration of the influence.
[0079]
For this reason, in some cases, as shown in FIG. 28, the weighting function may be a polygonal line function. The example of FIG. 28 shows an example in which the slope of the primary weighting function is changed near the halftone. The point at which the inclination is changed is not necessarily limited to the vicinity of the halftone, but is preferably determined according to the imaging conditions, the gamma characteristics of the liquid crystal, and the like.
[0080]
Alternatively, as shown in FIG. 29, the weighting function may be composed of a quadratic or higher order function. In the weighting function of FIG. 29, the weighting coefficient changes monotonously according to the gradation value, and the amount of change in the weighting coefficient near the halftone is reduced.
[0081]
In this way, the uneven pattern imaging for the subtraction process as shown in FIG. 19 is not frequently performed every time, but only needs to be performed once at the time of display device shipment or system setup by the user.
[0082]
The unevenness removal process described above is performed individually for each color of RGB, and then the captured image is displayed.
[0083]
As described above, in this embodiment, the unevenness component included in the multi-gradation data obtained by imaging with the sensor is removed based on the above-described equation (1). Imaging data that is not affected by this is obtained.
[0084]
In the embodiment described above, in the display device that performs color display by switching the emission color of the backlight from red → blue → green at a predetermined cycle without providing a color filter, an image is displayed for each pixel as shown in FIG. Although an example in which a capture sensor is provided has been described, a color filter may be provided in each pixel, and sensors may be arranged for one color or two colors among the three colors (RGB) constituting one pixel.
[0085]
For example, FIG. 30 shows an example in which a sensor is provided only for green among three colors constituting one pixel. The circuit configuration for one pixel in this case is represented by a circuit diagram as shown in FIG.
[0086]
An image capture sensor 33 is connected to the pixel TFT 31 corresponding to green via an initialization TFT 35. However, a pixel capture sensor is not connected to the pixel TFT corresponding to red and the pixel TFT corresponding to blue.
[0087]
With the configuration as shown in FIG. 31, the number of image capture sensors can be reduced, the structure can be simplified, and the aperture ratio when performing liquid crystal display can be improved.
[0088]
【The invention's effect】
As described above in detail, according to the present invention, the unevenness included in the first multi-tone data that is the imaging result of the subject is detected using the second multi-tone data that is the imaging result of the reference pattern. Because of the adjustment, multi-tone data without unevenness can be obtained. For this reason, even if unevenness occurs in the imaging result due to the characteristics of the imaging unit, the unevenness can be offset.
[Brief description of the drawings]
FIG. 1 is a block diagram showing an overall configuration of a display device according to the present invention.
FIG. 2 is a block diagram showing a circuit formed on the LCD substrate 1;
3 is a detailed circuit diagram for one pixel of the pixel array section 21. FIG.
FIG. 4 is a layout diagram for one pixel on a glass substrate.
FIG. 5 is a diagram for explaining a method of capturing an image.
FIG. 6 is a block diagram showing an internal configuration of the image processing IC 5;
FIG. 7 is a block diagram showing an example of the internal configuration of LCDC2.
FIG. 8 is a block diagram showing an internal configuration of a conventional LCDC2.
FIG. 9 is a flowchart showing a processing procedure when an image is captured by the LCDC 2;
FIG. 10 is a diagram illustrating a sequential addition method.
11 is a diagram showing signal exchange between the signal line driving circuit 22, the scanning line driving circuit 23, the sensor control circuit 24, and the signal processing output circuit 25 on the LCD substrate 1, and the LCDC 2 and the baseband LSI 3. FIG.
FIG. 12 is a block diagram showing a detailed configuration of a glass substrate.
FIG. 13 illustrates an operation of a display device according to an embodiment of the present invention.
FIG. 14 is a timing chart during normal display.
FIG. 15 is a timing chart when the sensor 33 is precharged and imaged;
FIG. 16 is a timing chart when the sensor 33 outputs image data.
FIG. 17 is a flowchart showing the processing operation of LCDC2.
FIG. 18 is a detailed flowchart of the process in step S3 of FIG.
FIG. 19 is a diagram showing a reproduced captured image of a white plate.
FIG. 20 is a diagram showing a captured image of an actual imaging object.
FIG. 21 is a diagram showing changes in gradation values of multi-gradation data obtained by imaging a white plate.
22 is a diagram showing a change in gradation value of the captured image shown in FIG. 20;
FIG. 23 is a view showing an image from which unevenness is removed from FIG. 20;
FIG. 24 is a diagram showing a change in gradation value corresponding to FIG.
FIG. 25 is a diagram illustrating a relationship between a gradation value of an imaging object and a gradation error of multi-gradation data obtained by imaging the imaging object.
FIG. 26 is a diagram illustrating a weighting function including a linear function.
FIG. 27 is a diagram illustrating nonlinear gamma characteristics.
FIG. 28 is a diagram illustrating a weighting function including a broken line.
FIG. 29 is a diagram illustrating a weighting function including a multi-order function.
FIG. 30 is a diagram illustrating an example in which a sensor is provided only in green among three colors constituting one pixel.
FIG. 31 is a circuit diagram showing a circuit configuration of one pixel.
[Explanation of symbols]
1 LCD board
2 LCDC
3 Baseband LSI
4 Camera
5 Image processing IC
6 transceiver
7 Power supply circuit
11 CPU
12 Main memory
13 MPEG processing section
14 DRAM
15 Control unit
16 frame memory
21 Pixel array section
22 Signal line drive circuit
23 Scanning line drive circuit
24 Sensor control circuit
25 Signal processing output circuit
31 pixel TFT
32 Display control TFT
33 Image capture sensor
34 SRAM
35 Initialization TFT

Claims (6)

Display elements in pixels formed near intersections of signal lines and scanning lines arranged in rows and columns, and
An imaging unit provided at least one for each of the display elements, each for imaging a predetermined range of a subject;
An initial multi-grayscale data storage unit for storing first multi-grayscale data corresponding to the imaging result of the imaging unit;
A reference pattern storage unit that stores second multi-gradation data obtained by imaging a predetermined reference pattern by the imaging unit;
An unevenness adjustment data generating unit that generates third multi-gradation data in which display unevenness of the first multi-gradation data is adjusted based on the first and second multi-gradation data,
The unevenness adjustment data generation unit
For each pixel, a difference calculation unit that calculates a difference between the second multi-gradation data and the gradation average value of the second multi-gradation data for one screen;
A weight calculation unit that calculates a weighting coefficient corresponding to the first multi-gradation data for unevenness adjustment set for each pixel for each pixel;
For each pixel, a non-uniformity adjustment value calculation unit that calculates the non-uniformity adjustment value by multiplying the corresponding weighting coefficient by the difference,
A display device, comprising: a non-uniformity canceling unit that generates the third multi-gradation data by subtracting the non-uniformity adjustment value from the first multi-gradation data for each pixel.   The display device according to claim 1, wherein the weight calculation unit calculates the weighting coefficient that changes monotonously according to a gradation value of the first multi-gradation data.   The weight calculation unit calculates the weighting coefficient based on a linear function indicating a correlation between a gradation value of the first multi-gradation data and the weighting coefficient. 2. The display device according to 2.   The weight calculation unit calculates the weighting coefficient based on a monotone change multi-order function indicating a correlation between a gradation value of the first multi-gradation data and the weighting coefficient. Item 4. The display device according to any one of Items 1 to 3.   The weight calculation unit makes the amount of change in the weighting factor for a gradation value near a halftone smaller than the amount of change in the weighting factor for a gradation value other than a halftone. The display apparatus in any one of.   6. The display device according to claim 1, wherein the absolute value of the weighting coefficient is maximum when the weight is black.

Images